When reading a disk file in ECL, the layout of the file is specified in the ECL code. This allows the code to be compiled to access the data very efficiently, but can cause issues if the file on disk is actually using a different layout.
In particular, it can present a challenge to the version control process, if you have ECL queries that are being changed to add functionality, but which need to be applied without modification to data files whose layout is changing on a different timeline.
There has been a partial solution to this dilemma available in Roxie for index files--the ability to apply runtime translation from the fields in the physical index file to the fields specified in the index. However, that has significant potential overhead and is not available for flat files or on Thor. This feature supports flat files and Thor files.
A new feature, added in the HPCC Systems 6.4.0 release, allows file resolution to be performed at compile time, which provides the following advantages:
Code changes can be insulated from file layout changes - you only need to declare the fields you actually want to use from a datafile.
File layout mismatches can be detected sooner.
The compiler can use information about file sizes to guide code optimization decisions.
There are two language constructs associated with this feature:
Using a LOOKUP attribute on DATASET or INDEX declarations.
Using a LOOKUP attribute in a RECORDOF function.
Adding the LOOKUP attribute to a DATASET declaration indicates that the file layout should be looked up at compile time:
myrecord := RECORD
STRING field1;
STRING field2;
END;
f := DATASET('myfilename', myrecord, FLAT);
// This will fail at runtime if file layout does not match myrecord
f := DATASET('myfilename', myrecord, FLAT, LOOKUP);
// This will automatically project from the actual to the requested layout
If we assume that the actual layout of the file on disk is:
myactualrecord := RECORD STRING field1; STRING field2; STRING field3; END;
Then the effect of the LOOKUP attribute will be as if your code was:
actualfile := DATASET('myfilename', myactualrecord, FLAT);
f := PROJECT(actualfile, TRANSFORM(myrecord, SELF := LEFT; SELF := []));
Fields that are present in both record structures are assigned across, fields that are present only in the disk version are dropped and fields that are present only in the ECL version receive their default value (a warning will be issued in this latter case).
There is also a compiler directive that can be used to specify translation for all files:
#OPTION('translateDFSlayouts',TRUE);The LOOKUP attribute accepts a parameter (TRUE or FALSE) to allow easier control of where and when you want translation to occur. Any Boolean expression that can be evaluated at compile time can be supplied.
When using the #OPTION for translateDFSlayouts, you may want to use LOOKUP(FALSE) to override the default on some specific datasets.