Once you have established standard ECL processes that you know you need to perform regularly, you can begin to make those processes automated. Doing this eliminates the need to remember the order of processes, or their periodicity.
One form of automation typically involves launching MACROs with the ECLPlus application. By using MACROs, you can have standard processes that operate on different input each time, but produce the same result. Since ECLPlus is a command-line application, its use can be automatically launched in many different ways -- DOS Batch files, from within another application, or ...
Here's an example. This MACRO (contained in DeclareData.ECL) takes two parameters: the name of a file, and the name of a field in that file to produce a count of the unique values in that field and a crosstab report of the number of instances of each value.
EXPORT MAC_CountFieldValues(infile,infield) := MACRO
// Create the count of unique values in the infield
COUNT(DEDUP(TABLE(infile,{infile.infield}),infield,ALL));
// Create the crosstab report
#UNIQUENAME(r_macro)
%r_macro% := RECORD
infile.infield;
INTEGER cnt := COUNT(GROUP);
END;
#UNIQUENAME(y_macro)
%y_macro% := TABLE(infile,%r_macro%,infield,FEW);
OUTPUT(CHOOSEN(%y_macro%,50000));
ENDMACRO;
By using #UNIQUENAME to generate all the attribute names, this MACRO can be used multiple times in the same workunit. You can test the MACRO through the ECL IDE program by executing a query like this in the ECL Builder window:
IMPORT ProgrammersGuide AS PG; PG.DeclareData.MAC_CountFieldValues(PG.DeclareData.Person.file,gender);
Once you've throughly tested the MACRO and are certain it works correctly, you can automate the process by using ECLplus.
Install the ECLplus program in its own directory on the same PC that runs the ECL IDE, and create an ECLPLUS.INI file in the same folder with the correct settings to access your cluster (see the Command Line ECL section of the Client Tools PDF). Then you can open a Command Prompt window and run the same query from the command line like this:
C:\eclplus>eclplus ecl=$ProgGuide.MAC_CountFieldValues(ProgrammersGuide.DeclareData.Person.File,gender)
Notice that you're using the ecl= command line option and not the $Module.Attribute option. This is the "proper" way to make a MACRO expand and execute through ECLplus. The $Module.Attribute option is only used to execute ECL Builder window queries that have been saved as attributes in the repository (Builder Window Runnable--BWR code) and won't work with MACROs.
When the MACRO expands and executes, you get a result that looks like this in your Command Prompt window:
Workunit W20070118-145647 submitted [Result 1] Result_1 2 [Result_2] gender cnt F 500000 M 500000
You can re-direct this output to a file by using the output="filename" option on the command line, like this:
C:\eclplus>eclplus ecl=$ProgGuide.MAC_CountFieldValues( ProgrammersGuide.DeclareData.Person.File, gender)
output="MyFile.txt"This will send the output to the "MyFile.txt" file on your local PC. For larger output files, you'll want to have the OUTPUT action in your ECL code write the result set to disk in the supercomputer then de-spray it to your landing zone (you can use the Standard Library's File.Despray function to do this from within your ECL code).
Another automation option is to generate a text file containing the ECL code to execute, then execute that code from the command line.
For example, you could create a file containing this:
IMPORT ProgrammersGuide AS PG; PG.DeclareData.MAC_CountFieldValues(PG.DeclareData.Person.file,gender); PG.DeclareData.MAC_CountFieldValues(PG.DeclareData.person.File,state)
These two MACRO calls will generate the field ordinality count and crosstab report for two fields in the same file. You could then execute them like this (where "test.ECL" is the name of the file you created):
C:\eclplus>eclplus @test.ecl
This will generate similar results to that above.
The advantage this method has is the ability to include any necessary "setup" ECL code in the file before the MACRO calls, like this (contained in RunText.ECL):
IMPORT ProgrammersGuide AS PG;
MyRec := RECORD
STRING1 value1;
STRING1 value2;
END;
D := DATASET([{'A','B'},
{'B','C'},
{'A','D'},
{'B','B'},
{'A','C'},
{'B','D'},
{'A','B'},
{'C','C'},
{'C','D'},
{'A','A'}],MyRec);
PG.DeclareData.MAC_CountFieldValues(D,Value1)
PG.DeclareData.MAC_CountFieldValues(D,Value2)
So that you get a result like this:
C:\eclplus>eclplus @test.ecl Workunit W20070118-145647 submitted [Result 1] result_1 3 [Result 2] value1 cnt C 2 A 5 B 3 [Result 3] result_3 4 [Result 4] value2 cnt D 3 C 3 A 1 B 3
How you create this text file is up to you. To fully automate the process you may want to write a daemon application that watches a directory (such as your HPCC Systems environment's landing zone) to detect new files dropped in (by whatever means) and generate the appropriate ECL code file to process that new file in some standard fashion (typically using MACRO calls), then execute it from ECLplus command line as described above. The realm of possibilities is endless.