Fri Dec 04, 2020 2:40 pm
Login Register Lost Password? Contact Us


#EXPAND Error -- Constant Expression Expected

Comments and questions related to the Enterprise Control Language

Wed Dec 05, 2012 6:43 am Change Time Zone

I am trying to extract all the records from one column in the datasets by using the function ITERATE to concatenate them to be part of the input of macro #EXPAND and apply it in the TABLE function to do sum/max. However, it gives the error msg as "Constant Expression Expected".

I am aware of the issue that
The "Constant expression expected" error is indicating that TABLE requires a constant expression as its RECORD structure and you're trying to build it "on the fly" and the compiler won't let you.

from viewtopic.php?t=535&p=2450

I am wondering if there is any alternative to accompolish this task. The code that gives the error is attached below:

DATA:
Code: Select all
EXPORT BureauLayoutSummary := MODULE

EXPORT Bureau_Layout:= RECORD

STRING  VariableName;
STRING Description;
STRING6 VarType;
STRING12 Format;
STRING1 Key_IND;
STRING20 MissingImputationBefore;
STRING2 MissingImputationAfter;
STRING AggregationFunc;

END;

EXPORT Bureau_Summary_Data :=
DATASET('~citi_bureau::burear_aggrfunc_test.csv',Bureau_Layout,CSV(HEADING(1)));
END;


Concatenate Function:

Code: Select all
RollUpKey := 'citi_cons_lnk,per_num';
SubFuncKeyWord :='SUM';
AggrVarNameSet := BureauLayoutSummary.Bureau_Summary_Data(AggregationFunc = SubFuncKeyWord);

STRING cancatVarnames(string RollUpKey, string SubFuncKeyWord) := function
   AggrVarNameSet cancatenateVars(AggrVarNameSet l, AggrVarNameSet r) := TRANSFORM
     pre_res := IF (l.VariableName = '','',l.VariableName+',');
     self.VariableName := pre_res + r.VariableName + '_' + SubFuncKeyWord +':=' + SubFuncKeyWord + '(GROUP,(REAL)'+r.VariableName+')';
       self := r;
   END;
   tempDS := ITERATE(AggrVarNameSet, cancatenateVars(LEFT, RIGHT));
   string tmp:= tempDS[count(tempDS)].VariableName;
    return tmp;
    return '{'+RollUpKey+','+tmp+'},'+ RollUpKey;
END;
STRING vars := cancatVarnames(RollUpKey, SubFuncKeyWord);
Dataset_aggr := DataPre.Aggregation_func(BUREAUDataImpt.RawData, vars);


The result from the function cancatVarnames returns a string I want but it fails and gives the error Constant Expression Expected. However it works when I copy the string vars and use the named string as an input of the #EXPAND() function below.

Call #EXPAND() macro:

Code: Select all
EXPORT Aggregation_func(InputData,tempVars) := FUNCTIONMACRO

   StrRollUp := '{'+RollUpKey+','+tempVars+'},'+ RollUpKey;
   Result := TABLE(InputData,#EXPAND(StrRollUp));
RETURN Result;
ENDMACRO;


Thanks!
dreamer1118
 
Posts: 3
Joined: Wed Dec 05, 2012 3:16 am

Wed Dec 05, 2012 8:27 pm Change Time Zone

dreamer,
The result from the function cancatVarnames returns a string I want but it fails and gives the error Constant Expression Expected. However it works when I copy the string vars and use the named string as an input of the #EXPAND() function below.
Yes, the error is telling you exactly what the problem is -- #EXPAND expects a string constant and you're giving it a non-constant string. Of course it works when you give it a string constant.
I am aware of the issue that

The "Constant expression expected" error is indicating that TABLE requires a constant expression as its RECORD structure and you're trying to build it "on the fly" and the compiler won't let you.

from viewtopic.php?t=535&p=2450
OK, then I have to ask you the same questions the person in that thread has not yet answered:
  • What exactly are you trying to accomplish?
  • Why are you writing this FUNCTIONMACRO to define a simple TABLE function?
Let me explain why I ask. It is my experience that file formats rarely change quickly. Certainly it is possible that, given that you receive a file from a particular source periodically that from period to period different programmers may introduce changes, but if the files are coming from any kind of automated process then their structure only changes when someone changes the process.

Therefore, since file formats rarely change, the best place to make any changes is in the code itself. Trying to generate new code from some "configuration file" simply introduces unnecessary complexity -- because it is exactly the same amount of work to change the one line of code as it is to change a "configuration file."

I am willing to be convinced that your scenario is an exception to my previous experience so please expand on the reasons you are trying to do things this way. :)

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1569
Joined: Wed Oct 26, 2011 7:40 pm

Thu Dec 06, 2012 2:28 am Change Time Zone

Thanks, Richard.

What we are trying to accomplish here is to extract information from an input data which contains a series of variable names that needs to do some aggregation (max,sum,ect.).

We are really new to ECL and the way we do it is to concatenate all the variables we need to do the aggregation function together, not one at a time. Say, we have five variables, v1, v2, v3, v4 ad v5 and group by key1. The output is expected to be like Key1, v1_sum, v2_sum, v3_sum, v4_sum, v5_sum.

The way we did is to use an iterate function to glue the variables together and then use #EXPAND() to transfer the information to TABLE function.

Hopefully I have made my question clearer. Thanks again for your help!
dreamer1118
 
Posts: 3
Joined: Wed Dec 05, 2012 3:16 am

Thu Dec 06, 2012 3:56 pm Change Time Zone

dreamer,
We are really new to ECL and the way we do it is to concatenate all the variables we need to do the aggregation function together, not one at a time. Say, we have five variables, v1, v2, v3, v4 ad v5 and group by key1. The output is expected to be like Key1, v1_sum, v2_sum, v3_sum, v4_sum, v5_sum.
OK, that's what I thought you were doing.

But the question remains -- why are you getting the "variable" names from another dataset and not simply writing them in the TABLE code? (BTW, in ECL there are no "variables," only definitions and fields in datasets, so these would normally be termed "fields").

Bottom line, to do what you want you just write this kind of code:
Code: Select all
rec := RECORD
  INTEGER key1;
  INTEGER v1;
  INTEGER v2;
  INTEGER v3;
  INTEGER v4;
  INTEGER v5;
END;

ds := DATASET([{1,1,1,1,1,1},{3,1,1,1,1,1},{2,1,1,1,1,1},
               {2,1,1,1,1,1},{3,1,1,1,1,1},{3,1,1,1,1,1}],rec);
                     
OutRec := RECORD
  ds.key1;
  s1 := SUM(GROUP,ds.v1);
  s2 := SUM(GROUP,ds.v2);
  s3 := SUM(GROUP,ds.v3);
  s4 := SUM(GROUP,ds.v4);
  s5 := SUM(GROUP,ds.v5);
END;                     

t := TABLE(ds,OutRec,key1);
t;
Note that, in order to work with the dataset (ds) you must define its RECORD structure, and you have named the fields here already. Since you have done this, the amount of work required to put those field names inside some other dataset that "drives" your TABLE function is the same amount of work required to simply write the TABLE function's RECORD structure with those field names or to write those field names into a constant string that you pass to your FUNCTIONMACRO like this:
Code: Select all
rec := RECORD
  INTEGER key1;
  INTEGER v1;
  INTEGER v2;
  INTEGER v3;
  INTEGER v4;
  INTEGER v5;
END;

ds := DATASET([{1,1,1,1,1,1},{3,1,1,1,1,1},{2,1,1,1,1,1},
               {2,1,1,1,1,1},{3,1,1,1,1,1},{3,1,1,1,1,1}],rec);
                     
rec2 := RECORD
  INTEGER key2;
  INTEGER f1;
  INTEGER f2;
  INTEGER f3;
  INTEGER f4;
  INTEGER f5;
END;
ds2 := DATASET([{4,1,1,1,1,1},{6,1,1,1,1,1},{5,1,1,1,1,1},
               {5,1,1,1,1,1},{6,1,1,1,1,1},{6,1,1,1,1,1}],rec2);                     

FM_GenXtabSUMs (InDS,OutRec,KeyField) := FUNCTIONMACRO
  RETURN TABLE(InDS,#EXPAND(OutRec),KeyField);
ENDMACRO;

PassRec1 := '{ds.key1,s1 := SUM(GROUP,ds.v1),s2 := SUM(GROUP,ds.v2),' +
            's3 := SUM(GROUP,ds.v3),s4 := SUM(GROUP,ds.v4),' +
                  's5 := SUM(GROUP,ds.v5)}';
FM_GenXtabSUMs(ds,PassRec1,Key1);

PassRec2 := '{ds2.key2,s1 := SUM(GROUP,ds2.f1),s2 := SUM(GROUP,ds2.f2),' +
            's3 := SUM(GROUP,ds2.f3),s4 := SUM(GROUP,ds2.f4),'+
                  's5 := SUM(GROUP,ds2.f5)}';
FM_GenXtabSUMs(ds2,PassRec2,Key2);
Passing the constant string keeps #EXPAND happy.

If there is some over-arching reason why you must get those field names from an external configuration file, then you should look at writing an external ECL code-generation program that reads your configuration file and generates the ECL code for you, which you could then launch with the command-line tool ECL.EXE (documented in the Client Tools PDF).

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1569
Joined: Wed Oct 26, 2011 7:40 pm

Fri Dec 07, 2012 3:55 am Change Time Zone

Thanks, Richard. I got your point through your instructive explanation.
And you are right that we have to use the field names as the input since we would like to make the function more flexible so that it can be further connect to user interface and let the clinet to make the decision on which fields to group.

We will refer to ECL code-generation program to see if it can solve this issue. Thanks again!


Lu
dreamer1118
 
Posts: 3
Joined: Wed Dec 05, 2012 3:16 am

Mon Dec 10, 2012 4:11 pm Change Time Zone

Lu,
And you are right that we have to use the field names as the input since we would like to make the function more flexible so that it can be further connect to user interface and let the clinet to make the decision on which fields to group.
OK, since the object is to allow end-users to select which fields they want to see these calculations on, then let me suggest a more HPCC-style alternative.

HPCC has three major components: Thor, Roxie, and ECL:

  • Thor is designed to be a "back office" tool. It allows you to solve your big data problems through "brute force" massive parallelism, so it is essentially a Big Data Supercomputer. It is designed to work with massive amounts of data, doing one job at a time, and prepare all your data for use by end-user queries.
  • Roxie is designed to service end-user queries. I handles thousands of concurrent transactions, delivering results to end-users "on demand" by using massive parallelism.
  • ECL is the common programming language for the two cluster types.
So the way HPCC is designed to work is to use Thor to do all your ETL and data preparation work, and Roxie to deliver the results to end-users. End-users don't send queries to Thor, only ECL developers do that. End-users only interface with pre-defined queries on Roxie (usually through some GUI that you've built to allow the interaction).

Therefore, given that you want your end-users to select which fields they are interested in calculating on, I would suggest that you simply use Thor to pre-calculate ALL the values that end-users might want to see from ALL your data. Then you just structure your Roxie queries to deliver only the set of fields that the end-user specifies for each given query. This way you're working with the system's design and not against it.

BTW, this kind of process is exactly what we teach in our ECL classes that you can sign up for here: http://hpccsystems.com/community/training-events/training

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1569
Joined: Wed Oct 26, 2011 7:40 pm


Return to ECL

Who is online

Users browsing this forum: Google [Bot] and 2 guests

cron