Dynamic Layout using a dataset

[SChatman85]

Hi,

I am trying to build a layout definition using information from within a dataset.

The below code should hopefully illustrate what I am trying to achieve, although it is in itself quite a messy way of trying to achieve it.

The code runs and produces the layout definition I expect - which is governed by the TRUE/FALSE filter in the profile dataset, but when I try and use it I run into the error:

Error: Constant expression expected (42, 12), 2071,

Is anyone able to either:
provide a better approach for achieving the layout
or, help resolve the above error

Thanks

Code: Select all

Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);
DS;

profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;

profile := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',TRUE},
                    {   'TITLE',FALSE}
                   ], profile_layout
                  );
profile;

temp_lay := RECORD
  profile;
  STRING ecl_string;
END;

profile(field_include=TRUE);
temp_profile := PROJECT(profile(field_include=TRUE), TRANSFORM(temp_lay, SELF := LEFT; SELF := [];));

temp_lay t_field_append(temp_profile L, temp_profile R) := TRANSFORM
  SELF.ecl_string := L.ecl_string + 'TYPEOF(DS.' + R.fieldname + ') ' + R.fieldname + ';';
  SELF := [];
END;

test := ITERATE(temp_profile, t_field_append(LEFT, RIGHT));
test;
output_layout := SORT(test, -LENGTH(ecl_string))[1];
OUTPUT('new_layout := RECORD ' + output_layout.ecl_string + ' END;');

/* new_layout := RECORD
    #EXPAND(output_layout.ecl_string)
   END;
   
   ds_new := PROJECT(DS, TRANSFORM(new_layout, SELF := LEFT;));
   ds_new;
*/

[rtaylor]

SChatman85,

In order to provide the best answer, I first need to know what exactly you're trying to accomplish. IOW, what problem are you trying to solve here?

HTH,

Richard

[SChatman85]

Hi Richard,

Thanks for the reply.

I have a dataset which acts as a configuration file (profile), in which there is a BOOLEAN field that can be set to TRUE or FALSE.

This configuration will be the layout of a logical file containing data that I need to carry out actions on.

What I want to do is change that logical file, removing all fields which are set to FALSE in the configuration.


My plan was to take all the rows in the profile dataset, where the value is TRUE, then take the field_name attribute and make that a layout I could then use within a PROEJECT/TRANSFORM.

Hope that all makes sense!

Stuart

[rtaylor]

Stuart,

OK, here's how I would do that:

Code: Select all

profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;
profile1 := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',TRUE},
                    {'TITLE',FALSE}
                   ], profile_layout
                  );
profile2 := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',FALSE},
                    {'TITLE',TRUE}
                   ], profile_layout
                  );

Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);

New_layout(DATASET(profile_layout) P) := RECORD
  IFBLOCK(P(fieldname='ID')[1].field_include=TRUE)
    STRING ID;
  END;
  IFBLOCK(P(fieldname='PREVIOUS_ID')[1].field_include=TRUE)
    STRING PREVIOUS_ID;
  END;
  IFBLOCK(P(fieldname='TITLE')[1].field_include=TRUE)
    STRING TITLE;
  END;
END;

PROJECT(DS,New_Layout(Profile1));
PROJECT(DS,New_Layout(Profile2));

Using an IFBLOCK structure around each field makes every field conditional, and passing the profile to use as a parameter to the new RECORD structure allows you to just use a simple PROJECT to the new structure. Notice the two results are different given different passed profiles.

HTH,

Richard

[SChatman85]

Thanks for the suggestion Richard.

IFBLOCK was not something I was aware of so will definitely take a look at what that offers - however, in terms of a solution I didn't explain the scale of whet I would need to do fully.

The profile definition may have around 100-200 rows (to match our dataset layout) and then I will have 40-50 of these profiles, one for each of the datafiles we want to process.


I would need something that works 'dynamically' to generate this as it will not be possible to explicitly code for each scenario.

Thanks,

Stuart

[rtaylor]

Stuart,

OK, this should get you a little further down the road:

Code: Select all

profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;
profile1 := DATASET([
                    {'id',TRUE},
                    {'previous_id',TRUE},
                    {'title',FALSE}
                   ], profile_layout
                  );
profile2 := DATASET([
                    {'id',TRUE},
                    {'previous_id',FALSE},
                    {'title',TRUE}
                   ], profile_layout
                  );

Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);

ProfileProject(inds, profds) := FUNCTIONMACRO
  prof_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
  END;

  #DECLARE(outrec);
  #SET(outrec,'OutputRecord(DATASET(prof_layout) P) := RECORD\n');
  #EXPORTXML(Fred,inds);
  #FOR (Fred)
    #FOR (Field)
      #APPEND(outrec,'  IFBLOCK(P(fieldname=\'' +
                     %'{@label}'% + '\')[1].field_include=TRUE)\n');
      #IF(%'{@size}'% = '-15')
        #APPEND(outrec,'    ' + %'{@type}'% + ' ' + %'{@label}'% + ';\n');
      #ELSE
        #APPEND(outrec,'    ' + %'{@type}'% + %'{@size}'%
                       + ' ' + %'{@label}'% + ';\n');
      #END
      #APPEND(outrec,'  END;\n');
    #END;
  #END;
  #APPEND(outrec,'END;\n');
  %outrec%;           //generate the RECORD structure code
  RETURN PROJECT(inds,OutputRecord(profds));
ENDMACRO;

ProfileProject(DS,Profile1);
ProfileProject(DS,Profile2);

NB -- the Profile field names are now all lowercase, because that's the way they're returned by #EXPORTXML.

HTH,

Richard

[SChatman85]

Thanks for joining the dots for me on this one! I hadn't thought of carrying out the PROJECT inside the FM - I had been focusing on just trying to get the layout to carry out the PROJECT outside.

I had to make one small change to handle field type of boolean - then when I printed out the MACRO RECORD structure it looked good - however when I run the code I receive the error

Code: Select all

Error:    ‘ctx’ was not declared in this scope (156, 21), 6003, W20201006-152704.cpp

I believe this is related to C++ but have no idea how I begin to look at finding what the issue is here?

[rtaylor]

Stuart,

I think that one requires a JIRA ticket with full details of your code and which version(s) of ECL/HPCC you're using.

HTH,

Richard

[SChatman85]

Thanks - have raised IDE-997

[ghalliday]

I think the way of achieving what you want is to have a file with all the potential fields. (PROJECT the input file if it has fewer fields.) Then have a PROJECT which clears any fields that are specified in the configuration file.