Sat Aug 15, 2020 7:38 am
Login Register Lost Password? Contact Us


Internal output exceeds limit error

Comments and questions related to the Enterprise Control Language

Fri May 29, 2020 3:16 am Change Time Zone

Hi,
Please help me understand more about the below error.

Code: Select all
9   System error: 10099: Graph graph28[2111], workunitwrite[2114]: Dataset too large to output to workunit (limit is set to 10) megabytes, in result (name=auto1L), Master exception - caused by (10099, Dataset too large to output to workunit (limit is set to 10) megabytes, in result (name=auto1L))


Code is huge to post also when I try to run only part of the code to simulate the issue it doesn't error out so, attached the portion of graph

I am not using SET which I know could potentially exceed the limit.

It would be nice to know when does Internal OUTPUT happens and why not Store internal or spill.

I also read one thread https://track.hpccsystems.com/browse/HPCC-15315 which says passing in a huge dataset to a function could cause this. In my case I do pass a huge dataset to function but its not a streamed dataset.

Please help.

Thanks,
Balaji
Attachments
Error.jpg
(48.98 KiB) Not downloaded yet
Last edited by BalajiTS on Sat May 30, 2020 7:53 am, edited 1 time in total.
BalajiTS
 
Posts: 4
Joined: Fri May 29, 2020 2:50 am

Fri May 29, 2020 12:40 pm Change Time Zone

Balaji,
graph28[2111], workunitwrite[2114]
This part of the error message is telling exactly which activity in the graph is causing the problem (activity 2114 in the 2111 subgraph, which is in graph 28). So look at that activity and relate it back you your ECL code to see which line of code it is.

The problem is that the intermediate dataset produced by that line of ECL (whatever it is) is more than 10 megabytes of data, and the limit is set to 10 (the default). You can try changing the limit using #OPTION like this:
Code: Select all
#OPTION('outputLimit',20); //double the limit to 20 Mb
or you can just find another way of accomplishing the task.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1552
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jun 01, 2020 5:15 pm Change Time Zone

Thank you Richard.
I reached out to Dan offline and came to know that moving MySQL plugin call processing to slave might solve this issue and it worked!!
BalajiTS
 
Posts: 4
Joined: Fri May 29, 2020 2:50 am


Return to ECL

Who is online

Users browsing this forum: Bing [Bot] and 1 guest

cron