Thu Jul 19, 2018 5:24 am
Login Register Lost Password? Contact Us


PIPE memory pool exhausted

Comments and questions related to the Enterprise Control Language

Thu Mar 15, 2018 10:48 pm Change Time Zone

I have a simple python script I wrote that just returns 'hello world' via stdout. I am calling this script with as follows:

Code: Select all
rec := RECORD
STRING11 out;
END;

myPipe := PIPE(recordInput, 'python myPythonScript.py ', rec);


If the record is "STRING11" it works fine. If i change it to "STRING", to try and accept a variable length string, it returns this error: "Error: System error: 1301: Pool memory exhausted: ...". An 11 character length string doesn't seem that large.

I have also tried returning from python a C struct, which creates a different set of errors and only gets a memory exhaust error if the struct is over 100 long.

Anyone have this issue or have a link to more documentation on what exactly PIPE() is expecting. My end goal is to simply return a variable length (string, int, data, etc) to a record...

Thanks in advance
rken
 
Posts: 7
Joined: Thu Feb 08, 2018 6:30 pm

Fri Mar 16, 2018 1:35 pm Change Time Zone

rken,

I think your problem is that the internal respresentation of a variable-length STRING is different to that of a fixed-length STRINGn.

I suggest you try using a large fixed-length string as your return type (maybe STRING200 in this case) and see if that works for you. If it does, then your workaround is to understand what your maximize size return value is likely to be and define your STRINGn to something a bit larger.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1366
Joined: Wed Oct 26, 2011 7:40 pm

Fri Mar 16, 2018 1:40 pm Change Time Zone

Hello,
I agree with Richard's comment, but I also found a similar post regarding the memory pool with someone using embedded python.
https://hpccsystems.com/bb/viewtopic.php?f=14&t=4823

May or may not be related, but it might be worth it to post this to our JIRA Issue Tracker.

https://track.hpccsystems.com/secure/Dashboard.jspa

If it is true that variable length strings are not permitted it may be something to add to our documentation.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 975
Joined: Wed Jun 29, 2011 7:13 pm

Fri Mar 16, 2018 1:57 pm Change Time Zone

Also, did you try approaching this using embedded Python?

See the following topic for more information:

https://hpccsystems.com/training/documentation/ecl-language-reference/html/EMBED_Structure.html

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 975
Joined: Wed Jun 29, 2011 7:13 pm

Fri Mar 16, 2018 2:06 pm Change Time Zone

rken,

It also occurs to me that you could try making your Python code return your variable-length STRING in the format that HPCC expects it -- a leading 4-byte integer value indicating the length of the text prepended to the text itself, something like this:
Code: Select all
//this is obviously ECL, but you should get the idea:

  //hex representation of 11, in Little Endian format:
DATA4 LeadingLength := x'0B000000';

  //type transfer to STRING4 and concatenate to return string
ReturnStr := ((>STRING4<)LeadingLength) + 'Hello World';

Let me know if that works, please. :)

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1366
Joined: Wed Oct 26, 2011 7:40 pm

Fri Mar 16, 2018 8:13 pm Change Time Zone

1: I tried using a large fixed length string(and other types too) and they work as expected. It also doesn't fail if i use a STRINGn where: n % string.length() == 0, except it just returns 100 rows, slicing off whatever is after.

2 and 3: With EMBED(python), i can return, in python, a [bytearray] into a rec:= DATA d; END; with no problem, tested it up to a 7 mbit string. This indicates that there is something else besides the string that is communicated between Python and ECL via the plugin.
rken
 
Posts: 7
Joined: Thu Feb 08, 2018 6:30 pm

Fri Mar 16, 2018 8:19 pm Change Time Zone

rtaylor wrote:rken,

It also occurs to me that you could try making your Python code return your variable-length STRING in the format that HPCC expects it -- a leading 4-byte integer value indicating the length of the text prepended to the text itself, something like this:
Code: Select all
//this is obviously ECL, but you should get the idea:

  //hex representation of 11, in Little Endian format:
DATA4 LeadingLength := x'0B000000';

  //type transfer to STRING4 and concatenate to return string
ReturnStr := ((>STRING4<)LeadingLength) + 'Hello World';

Let me know if that works, please. :)

HTH,

Richard



Yes, this works! Exactly what I was looking for. Adding a 4 byte little endian, signed integer to the front of my python scripts stdout works like a charm.

ECL Record being returned by PIPE():
Code: Select all
rec := RECORD
  STRING output;
END;


Python Code:

Code: Select all
lorem = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam condimentum tincidunt quam et vulputate metus."
sOut = struct.pack('<i', len(lorem))
sys.stdout.write(sOut + lorem)


The above outputs a single record row with the lorem text. If one wants to output multiple rows in the record, simply have separate stdout calls acomplishes this (perhaps adding a std.flush() in between might increase stability)

Code: Select all
lorem = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam condimentum tincidunt quam et vulputate metus."
sOut = struct.pack('<i', len(lorem))

for _ in range(n):
     sys.stdout.write(sOut + lorem)


I think this would be helpful to put in the documentation for PIPE(): what exactly is expected if not CSV or XML.
rken
 
Posts: 7
Joined: Thu Feb 08, 2018 6:30 pm

Mon Mar 19, 2018 1:32 pm Change Time Zone

rken,

Glad that works for you! :)
I think this would be helpful to put in the documentation for PIPE(): what exactly is expected if not CSV or XML.
I agree. And since you're new to the Forum, I expect you may also be new to our bug tracking/feature request system: JIRA - https://track.hpccsystems.com.

You can use JIRA to make any reports you need to and you'll automatically be updated as the ticket makes its way from initial report, through developer discussion, and on to completion.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1366
Joined: Wed Oct 26, 2011 7:40 pm


Return to ECL

Who is online

Users browsing this forum: Google [Bot] and 0 guests