Sun Oct 25, 2020 10:51 am
Login Register Lost Password? Contact Us


Dynamic Layout using a dataset

Questions around writing code and queries

Wed Sep 23, 2020 3:42 pm Change Time Zone

Hi,

I am trying to build a layout definition using information from within a dataset.

The below code should hopefully illustrate what I am trying to achieve, although it is in itself quite a messy way of trying to achieve it.

The code runs and produces the layout definition I expect - which is governed by the TRUE/FALSE filter in the profile dataset, but when I try and use it I run into the error:

Error: Constant expression expected (42, 12), 2071,


Is anyone able to either:
provide a better approach for achieving the layout
or, help resolve the above error

Thanks

Code: Select all
Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);
DS;

profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;

profile := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',TRUE},
                    {   'TITLE',FALSE}
                   ], profile_layout
                  );
profile;

temp_lay := RECORD
  profile;
  STRING ecl_string;
END;

profile(field_include=TRUE);
temp_profile := PROJECT(profile(field_include=TRUE), TRANSFORM(temp_lay, SELF := LEFT; SELF := [];));

temp_lay t_field_append(temp_profile L, temp_profile R) := TRANSFORM
  SELF.ecl_string := L.ecl_string + 'TYPEOF(DS.' + R.fieldname + ') ' + R.fieldname + ';';
  SELF := [];
END;

test := ITERATE(temp_profile, t_field_append(LEFT, RIGHT));
test;
output_layout := SORT(test, -LENGTH(ecl_string))[1];
OUTPUT('new_layout := RECORD ' + output_layout.ecl_string + ' END;');

/* new_layout := RECORD
    #EXPAND(output_layout.ecl_string)
   END;
   
   ds_new := PROJECT(DS, TRANSFORM(new_layout, SELF := LEFT;));
   ds_new;
*/
SChatman85
 
Posts: 17
Joined: Mon Sep 02, 2019 2:13 pm

Wed Sep 23, 2020 7:15 pm Change Time Zone

SChatman85,

In order to provide the best answer, I first need to know what exactly you're trying to accomplish. IOW, what problem are you trying to solve here?

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1560
Joined: Wed Oct 26, 2011 7:40 pm

Thu Sep 24, 2020 7:46 am Change Time Zone

Hi Richard,

Thanks for the reply.

I have a dataset which acts as a configuration file (profile), in which there is a BOOLEAN field that can be set to TRUE or FALSE.

This configuration will be the layout of a logical file containing data that I need to carry out actions on.

What I want to do is change that logical file, removing all fields which are set to FALSE in the configuration.


My plan was to take all the rows in the profile dataset, where the value is TRUE, then take the field_name attribute and make that a layout I could then use within a PROEJECT/TRANSFORM.

Hope that all makes sense!

Stuart
SChatman85
 
Posts: 17
Joined: Mon Sep 02, 2019 2:13 pm

Thu Sep 24, 2020 12:23 pm Change Time Zone

Stuart,

OK, here's how I would do that:
Code: Select all
profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;
profile1 := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',TRUE},
                    {'TITLE',FALSE}
                   ], profile_layout
                  );
profile2 := DATASET([
                    {'ID',TRUE},
                    {'PREVIOUS_ID',FALSE},
                    {'TITLE',TRUE}
                   ], profile_layout
                  );

Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);

New_layout(DATASET(profile_layout) P) := RECORD
  IFBLOCK(P(fieldname='ID')[1].field_include=TRUE)
    STRING ID;
  END;
  IFBLOCK(P(fieldname='PREVIOUS_ID')[1].field_include=TRUE)
    STRING PREVIOUS_ID;
  END;
  IFBLOCK(P(fieldname='TITLE')[1].field_include=TRUE)
    STRING TITLE;
  END;
END;

PROJECT(DS,New_Layout(Profile1));
PROJECT(DS,New_Layout(Profile2));
Using an IFBLOCK structure around each field makes every field conditional, and passing the profile to use as a parameter to the new RECORD structure allows you to just use a simple PROJECT to the new structure. Notice the two results are different given different passed profiles.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1560
Joined: Wed Oct 26, 2011 7:40 pm

Thu Oct 01, 2020 10:25 am Change Time Zone

Thanks for the suggestion Richard.

IFBLOCK was not something I was aware of so will definitely take a look at what that offers - however, in terms of a solution I didn't explain the scale of whet I would need to do fully.

The profile definition may have around 100-200 rows (to match our dataset layout) and then I will have 40-50 of these profiles, one for each of the datafiles we want to process.


I would need something that works 'dynamically' to generate this as it will not be possible to explicitly code for each scenario.

Thanks,

Stuart
SChatman85
 
Posts: 17
Joined: Mon Sep 02, 2019 2:13 pm

Fri Oct 02, 2020 8:01 pm Change Time Zone

Stuart,

OK, this should get you a little further down the road:
Code: Select all
profile_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
END;
profile1 := DATASET([
                    {'id',TRUE},
                    {'previous_id',TRUE},
                    {'title',FALSE}
                   ], profile_layout
                  );
profile2 := DATASET([
                    {'id',TRUE},
                    {'previous_id',FALSE},
                    {'title',TRUE}
                   ], profile_layout
                  );

Actual_Lay := RECORD
  STRING ID;
  STRING PREVIOUS_ID;
  STRING TITLE;
END;

DS := DATASET([{'123','21', 'MR'}],Actual_Lay);

ProfileProject(inds, profds) := FUNCTIONMACRO
  prof_layout := RECORD
    STRING  fieldname;
    BOOLEAN field_include;
  END;

  #DECLARE(outrec);
  #SET(outrec,'OutputRecord(DATASET(prof_layout) P) := RECORD\n');
  #EXPORTXML(Fred,inds);
  #FOR (Fred)
    #FOR (Field)
      #APPEND(outrec,'  IFBLOCK(P(fieldname=\'' +
                     %'{@label}'% + '\')[1].field_include=TRUE)\n');
      #IF(%'{@size}'% = '-15')
        #APPEND(outrec,'    ' + %'{@type}'% + ' ' + %'{@label}'% + ';\n');
      #ELSE
        #APPEND(outrec,'    ' + %'{@type}'% + %'{@size}'%
                       + ' ' + %'{@label}'% + ';\n');
      #END
      #APPEND(outrec,'  END;\n');
    #END;
  #END;
  #APPEND(outrec,'END;\n');
  %outrec%;           //generate the RECORD structure code
  RETURN PROJECT(inds,OutputRecord(profds));
ENDMACRO;

ProfileProject(DS,Profile1);
ProfileProject(DS,Profile2);
NB -- the Profile field names are now all lowercase, because that's the way they're returned by #EXPORTXML.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1560
Joined: Wed Oct 26, 2011 7:40 pm

Tue Oct 06, 2020 2:28 pm Change Time Zone

Thanks for joining the dots for me on this one! I hadn't thought of carrying out the PROJECT inside the FM - I had been focusing on just trying to get the layout to carry out the PROJECT outside.

I had to make one small change to handle field type of boolean - then when I printed out the MACRO RECORD structure it looked good - however when I run the code I receive the error
Code: Select all
Error:    ‘ctx’ was not declared in this scope (156, 21), 6003, W20201006-152704.cpp


I believe this is related to C++ but have no idea how I begin to look at finding what the issue is here?
SChatman85
 
Posts: 17
Joined: Mon Sep 02, 2019 2:13 pm

Tue Oct 06, 2020 7:29 pm Change Time Zone

Stuart,

I think that one requires a JIRA ticket with full details of your code and which version(s) of ECL/HPCC you're using.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1560
Joined: Wed Oct 26, 2011 7:40 pm

Wed Oct 07, 2020 8:07 am Change Time Zone

Thanks - have raised IDE-997
SChatman85
 
Posts: 17
Joined: Mon Sep 02, 2019 2:13 pm

Mon Oct 12, 2020 3:30 pm Change Time Zone

I think the way of achieving what you want is to have a file with all the potential fields. (PROJECT the input file if it has fewer fields.) Then have a PROJECT which clears any fields that are specified in the configuration file.
ghalliday
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 197
Joined: Wed May 18, 2011 9:48 am

Next

Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest

cron