Mon Apr 22, 2019 4:54 pm
Login Register Lost Password? Contact Us


Request help using #GETDATATYPE in MACRO

Comments and questions related to the Enterprise Control Language

Wed Jan 16, 2019 10:52 am Change Time Zone

Hi,
I've been tieing myself in knots trying to dynamically construct a record structure given the #GETDATATYPE of a parameter to a FUNCTIONMACRO.
Original code:

EXPORT ConcatenateStringFields(_ds_,_fld_,_sep_) := FUNCTIONMACRO

_ds1_ := PROJECT(_ds_,TRANSFORM({STRING _fld_},SELF._fld_ := LEFT._fld_));

RETURN ROLLUP(_ds1_,TRUE,TRANSFORM({STRING _fld_},SELF._fld_ := LEFT._fld_ + _sep_ + RIGHT._fld_))[1]._fld_;

ENDMACRO;

This macro is already very useful, but, as you can tell from its name, its confined to be being used by STRINGs.
If the record types in bold could be dictated by the '_fld_' or '_sep_' parameter the MACRO would be even more useful.

A 'nice to have' would also be a check that the type of the '_fld_' matched the type of '_sep_' and ASSERT FAIL if not. (with useful message)

Thanks for your efforts in advance.
Yours
Allan
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 16, 2019 12:10 pm Change Time Zone

You don't really say what the problem is you are hitting with #getdatatype. That it doesn't work, or generates unusable results?

As an aside you will find AGGREGATE is likely to be more efficient than ROLLUP.
ghalliday
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 181
Joined: Wed May 18, 2011 9:48 am

Wed Jan 16, 2019 1:45 pm Change Time Zone

AH Well this is embarrassing, I recall nearly getting it working on Monday. (last thing late in day)
An example where it flattens UNICODE field in a DATASET.
I had to hard code 'arr.chr' in the FUNCTIONMACRO. So getting over that hurdle would be great. I actually was not getting a problem using #GETDATATYPE. I expect I could sort it out, but had incorrectly remembered not getting it working at all.
Code: Select all
UNICODE arrowarray := U'\u2190\u2191\u2192\u2193\u2196\u2197\u2198\u2199';
LL := 79;         // Line length

RDS := {UNICODE chr,BOOLEAN Changed};

RDS Create({UNICODE a} L) := TRANSFORM
    SELF.chr := arrowarray[(RANDOM() % 8)+1];
    SELF.Changed := FALSE;
END;

arr := NORMALIZE(DATASET([{U'\u2190'}],{UNICODE a}),LL*60,Create(LEFT));


ConcatenateFields(_ds_,_fld_,_sep_) := FUNCTIONMACRO

#DECLARE(field);
#SET(field, 'arr.chr');
#DECLARE(fieldtype);
#SET(fieldtype, #GETDATATYPE(%field%) ); 
   _ds1_ := PROJECT(_ds_,TRANSFORM({%fieldtype% _fld_},SELF._fld_ := LEFT._fld_));

  RETURN ROLLUP(_ds1_,TRUE,TRANSFORM({%fieldtype% _fld_},SELF._fld_ := LEFT._fld_ + _sep_ + RIGHT._fld_))[1]._fld_;

ENDMACRO;

d := DATASET([{ConcatenateFields(arr,chr,U''),TRUE}],{UNICODE str,BOOLEAN Changed});
d;


As to using AGGREGATE, this macro was written years back by Rob, there was a long discussion then on the merits on AGGREGATE over ROLLUP, I think it was decided that ROLLUP was best.
Apologies for wasting people time.
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 16, 2019 1:52 pm Change Time Zone

replacing
Code: Select all
#SET(field, 'arr.chr');

with
Code: Select all
#SET(field, _ds_._fld_);


fails with 'constant expression expected.

All's well with:
Code: Select all
#SET(field, #TEXT(_ds_._fld_));


so got the whole thing working.
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 16, 2019 2:08 pm Change Time Zone

for completeness, here it the working version, with type checking in place.
Code: Select all
ConcatenateFields(_ds_,_fld_,_sep_) := FUNCTIONMACRO

#DECLARE(Seperator);
#SET(Seperator, #TEXT(_sep_));
#DECLARE(SeperatorType);
#SET(SeperatorType, #GETDATATYPE(%Seperator%) ); 
#DECLARE(field);
#SET(field, #TEXT(_ds_._fld_));
#DECLARE(fieldtype);
#SET(fieldtype, #GETDATATYPE(%field%) ); 
   _ds1_ := PROJECT(_ds_,TRANSFORM({%fieldtype% _fld_},SELF._fld_ := LEFT._fld_));

   BaseType(STRING ty) := REGEXREPLACE('[^[:alpha:]]',ty,'');
   chk := ASSERT(BaseType(%'fieldtype'%) = BaseType(%'SeperatorType'%),'Type of \'Seperator\' ('+%'SeperatorType'%+') must match the type of field being concaternated ('+%'fieldtype'%+').',FAIL);
  RETURN WHEN(ROLLUP(_ds1_,TRUE,TRANSFORM({%fieldtype% _fld_},SELF._fld_ := LEFT._fld_ + _sep_ + RIGHT._fld_))[1]._fld_,chk);

ENDMACRO;
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 16, 2019 2:21 pm Change Time Zone

Actually the test in the ASSERT, may need further work. No time at the moment.
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 16, 2019 2:34 pm Change Time Zone

Allan,

Here's what I get when I run your code:
Code: Select all
ds := DATASET([{'1',1},{'2',2},{'3',3}],{STRING s, UNSIGNED l});

ConcatenateFields(ds,s,'-'); //works just fine: 1-2-3
ConcatenateFields(ds,l,'-'); //Error: Can not assign String to Integer (field SELF.l)
ConcatenateFields(ds,l,0);   //Error: Type of 'Seperator' (integer8) must match the
                             //type of field being concaternated (unsigned8).

Since concatenation is the mission here, you'll need to cast any non-STRING datatypes to STRING to achieve that.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1444
Joined: Wed Oct 26, 2011 7:40 pm

Wed Jan 16, 2019 5:27 pm Change Time Zone

Thanks Richard,

I've rushed and I'll have to do a lot more work on this to mak it production ready.
Raised a 'question' with the core team:
https://track.hpccsystems.com/browse/HPCC-21330
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Wed Jan 30, 2019 3:30 pm Change Time Zone

I had a similar problem awhile back and still have the code, so I thought I'd share. I don't know if it addresses all of your concerns, but it perhaps it is close.

Code: Select all
ConcatFields(inFile, outFieldType, outField, fieldListStr, delim = '\'\'') := FUNCTIONMACRO
    #UNIQUENAME(myDelim);
    #IF(delim != '')
        #SET(myDelim, '\'' + delim + '\' + ')
    #ELSE
        #SET(myDelim, '')
    #END
    #UNIQUENAME(rhs);
    #SET(rhs, REGEXREPLACE(',', REGEXREPLACE('(\\w+)', TRIM((STRING)fieldListStr, ALL), '(' + #TEXT(outFieldType) + ')LEFT.$1'), ' + ' + %'myDelim'%));

    // RETURN %'rhs'%;
    RETURN PROJECT
        (
            inFile,
            TRANSFORM
                (
                    {
                        RECORDOF(inFile),
                        outFieldType  outField
                    },
                    SELF.outField := %rhs%,
                    SELF := LEFT
                )
        );
ENDMACRO;

//------------------------------------------------------------------------------

DataRec := RECORD
    QSTRING     fname;
    STRING1     mname_initial;
    STRING      lname;
    UNSIGNED1   age;
END;

ds := DATASET
    (
        [
            {'Dan', 'S', 'Camper', 53}
        ],
        DataRec
    );

ConcatFields(ds, STRING, myOut, 'fname, mname_initial, lname', ',');
ConcatFields(ds, QSTRING, myOut, 'fname, age', ':');
ConcatFields(ds, UTF8, myOut, 'age, lname');
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 554
Joined: Tue Oct 18, 2011 4:45 pm

Thu Jan 31, 2019 8:24 pm Change Time Zone

Hi,

That's really nice, and I'm going to keep that ECL in my pocket. As to my particular issue, its different, in that I'm concatenating a single field from all records in a dataset into a single STRING (with optional delimiters)

Yours

Allan
Allan
 
Posts: 353
Joined: Sat Oct 01, 2011 7:26 pm

Next

Return to ECL

Who is online

Users browsing this forum: No registered users and 0 guests