Wed Aug 15, 2018 4:38 pm
Login Register Lost Password? Contact Us


Why is it that I can output a table but not write into file?

Questions around writing code and queries

Mon Jan 22, 2018 7:19 pm Change Time Zone

Hi there!

I've been trying to parse out some XML information and had to brut force a string merge so that I can get all the fields I want into 1 table. The issue I am having is when I write OUTPUT(filename); it is fine. When I write OUTPUT(filename,,'~filename',THOR); The code will brake and tell me I have a parsing error. This also happens when I am trying to join that dataset onto another dataset. Why is this happening?

I am so stuck! Help! Please!!

Katy

Attaching workunits (code is too long) & I ran this in Alpha Prod environment!

Here is a working WUID
Here is a non working WUID
Last edited by KatyChow on Mon Jan 22, 2018 8:50 pm, edited 1 time in total.
KatyChow
 
Posts: 25
Joined: Mon Jun 02, 2014 3:16 pm

Mon Jan 22, 2018 8:14 pm Change Time Zone

Katy,

Try it without the THOR option on your OUTPUT, like this:
Code: Select all
OUTPUT(filename,,'~filename');

The default output file type is already "Thor" (AKA: FLAT files) so the THOR option on OUTPUT specifies to "OUTPUT Workunit Files" which are documented here:https://hpccsystems.com/training/documentation/ecl-language-reference/html/OUTPUT.html%23OUTPUT_Workunit_Files

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1368
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jan 22, 2018 8:42 pm Change Time Zone

Hi Richard,

No such luck. I'm getting a different error now... .

Here's the WUID
Attaching trucated code:
Code: Select all
O(x, num = 100) := MACRO
  OUTPUT(CHOOSEN(x, num), NAMED(#TEXT(x)));
ENDMACRO;

OC(x, num = 100) := MACRO
  OUTPUT(CHOOSEN(x, num), NAMED(#TEXT(x)));
  OUTPUT(COUNT(x), NAMED('Num_' + #TEXT(x)));
ENDMACRO;

OA(x) := MACRO
  OUTPUT(x, NAMED(#TEXT(x)), ALL);
ENDMACRO;

C(x) := MACRO
  OUTPUT(COUNT(x), NAMED('Num_' + #TEXT(x)));
ENDMACRO;

rec1 := RECORD
  string transaction_id;
  string11 product_id;
  string19 date_added;
  string4 process_type;
  string8 processing_time;
  string10 vendor_code;
  string20 request_type;
  string20 product_version;
  string15 reference_number;
  string content_data{blob, maxlength(2000000)};
  string6 process_status;
END;

logs := dataset('~base::fcra_mbsi::inquiry_history::qa::id',rec1,thor)(date_added[1..10] >= '2017-12-01' AND date_added[1..10] <= '2017-12-31');
OC(logs);
s_logs := sort(logs,transaction_id);
OC(s_logs);

model_attr := logs(request_type = 'MODEL_ATTRIBUTES');
OC(model_attr);

pub_recs_boca0 := logs(request_type = 'FCRA_PUB_REC_RESP');
OC(pub_recs_boca0);

// need to string-a-fy the blob....
pub_recs_boca := project(pub_recs_boca0, TRANSFORM({recordof(pub_recs_boca0); string content_str;},



True_Scores := model_attr(REGEXFIND('<status_message>No Score</status_message>',content_data)=FALSE);
OC(True_Scores);

MV36 := True_Scores(REGEXFIND('<name>MV36</name>',content_data));
MX36 := True_Scores(REGEXFIND('<name>MX36</name>',content_data));
MNC2 := True_Scores(REGEXFIND('<name>MNC2</name>',content_data));

Models_ds := MV36+MX36+MNC2;
OC(Models_ds);

  boca_attr_layout := record
  string16 transaction_id;
   STRING fcra;
   STRING cb_allowed;
   STRING account;
   STRING seq;
        ...
END;


boca_attr_layout parse_boca_attr_blob (rec1 l) := transform
   self.transaction_id := l.transaction_id;
   content := l.content_data;   
    str_start := std.str.find(content,'<Attributes>');
    str_end := std.str.find(content,'</Attributes>');
    str_start2_find := content[str_end+13..];
    str_start2 := std.str.find(str_start2_find,'<Attributes>');
    str_end2 := std.str.find(str_start2_find,'</Attributes>');
   attr_string1 := content[str_start .. str_end-1];
    attr_string2 := str_start2_find[str_start2+12 .. str_end2+12];
    attr_string := attr_string1 + attr_string2;
   wrapped_attr_string := '<s>'+attr_string+'</s>';
   
   nv_pair := RECORD
    STRING50 Name {xpath('Name')};
    STRING   Value {xpath('Value'), maxlength(256)};
   END;
   temp_layout := record
     dataset(nv_pair) attr{xpath('Attributes/Attribute')};
    END;
   attr_table := fromxml(temp_layout,attr_string);
   
    SELF.fcra := attr_table.attr(name = 'fcra')[1].value;
    SELF.cb_allowed := attr_table.attr(name = 'cb_allowed')[1].value;
    SELF.account := attr_table.attr(name = 'account')[1].value;
    SELF.seq := attr_table.attr(name = 'seq')[1].value;
    ...
   END;
   
   boca_attr_parsed := project(pub_recs_boca,parse_boca_attr_blob(left));
   // OUTPUT(boca_attr_parsed,named('boca_atts_parsed'));
OUTPUT(boca_attr_parsed,,'~MVR::KC::Boca_parsedXML', EXPIRE(5));
Last edited by KatyChow on Mon Jan 22, 2018 8:50 pm, edited 1 time in total.
KatyChow
 
Posts: 25
Joined: Mon Jun 02, 2014 3:16 pm

Mon Jan 22, 2018 8:48 pm Change Time Zone

Katy,

Remember, this is a public forum, so many non-LN people will not be able to see your workunit. Therefore, please post the specific error you are now getting.

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1368
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jan 22, 2018 8:51 pm Change Time Zone

Oh! Oops! oKay!

Here it is
Error: System error: 1: Graph graph1[1], project[6]: SLAVE #91 [10.194.198.91:22000]: Error - end of stream "End of stream encountered whilst parsing" [file offset 68601]
>0.3272036910057068</Value></Attribute>G*ERROR*, - caused by (1, Error - end of stream "End of stream encountered whilst parsing" [file offset 68601]
>0.3272036910057068</Value></Attribute>G*ERROR*) (0, 0), 1,
Warning: Activity 6 created a complex helper class (21714) (2710, 22), 4538,
KatyChow
 
Posts: 25
Joined: Mon Jun 02, 2014 3:16 pm

Mon Jan 22, 2018 9:04 pm Change Time Zone

Katy,
"End of stream encountered whilst parsing" [file offset 68601]
This error basically says the code ran out of file before it found what it was looking for. IOW, this is most likely a data error.

I'd suggest taking a very close look at your input file(s) right around the 68601 byte position, or prior to.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1368
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jan 22, 2018 9:09 pm Change Time Zone

Hi Richard,

I guess I'm still confused about why this would run when I didn't output into a file?

Thanks!

Katy
KatyChow
 
Posts: 25
Joined: Mon Jun 02, 2014 3:16 pm

Mon Jan 22, 2018 9:28 pm Change Time Zone

Katy,
why this would run when I didn't output into a file?
Because in order to do a simple OUTPUT (IOW, "show me the first 100 records") the system didn't have to read enough records to encounter the data problem. But when you asked it to write all the result data to a file, it had to process all the data, and that's when it found the problem.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1368
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jan 22, 2018 9:32 pm Change Time Zone

Hi Richard!

Ah okay! I get it now!

Thanks!!

Katy
KatyChow
 
Posts: 25
Joined: Mon Jun 02, 2014 3:16 pm


Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest