Fri Aug 17, 2018 3:07 am
Login Register Lost Password? Contact Us


Memory pool exhausted when passing 2 datasets embed

Comments and questions related to the Enterprise Control Language

Fri Sep 29, 2017 2:48 pm Change Time Zone

Below is the code. Currently, my embedded python doesn't do anything. The ECL reads in 2 large datasets and passes both to the embedded python, MatrixMultiply. The sizes of the 2 datasets are 1,600,040,000 and 1,600,080,000. I get the following error:
Code: Select all
Error: System error: 1301: Memory pool exhausted: pool id 4194314 (1216 pages) exhausted, requested 1 (in Disk Read G22 E23)

Code: Select all
import python;

rec0 := RECORD
  REAL cell;
END;

rec := RECORD
DATASET(rec0) arow;
END;

DATASET(rec) MatrixMultiply(DATASET(rec) A, unsigned nrowsA, unsigned ncolsA,DATASET(rec) B, unsigned nrowsB, unsigned ncolsB) := embed(Python)
    import numpy as np
    import re
    return A

endembed;

A:=DATASET('~hthor::tlh::AMatrix',rec,THOR);
B:=DATASET('~hthor::tlh::BMatrix',rec,THOR);
NRowsA:=COUNT(A);
NColsA:=COUNT(A[1].arow);
NRowsB:=COUNT(B);
NColsB:=COUNT(B[1].arow);

MatrixMultiply(A,NRowsA,NColsA,B,NRowsB,NColsB);

I'm running this on hthor. I'm working on a machine that has 15GB of memory. So, I should have plenty of memory. Both datasets should fit entirely in memory. I want to make changes to my environment.xml file so more memory will be available to my workunit. I have added these 2 parameters to both EclAgentProcess and ThorCluster. But, I'm still getting the error.
Code: Select all
defaultMemoryLimit="10000000000"
totalMemoryLimit="10000000000"

Ang help would be most appreciated.
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Fri Sep 29, 2017 2:55 pm Change Time Zone

Tim,

Just curious -- since your embedded Python "does nothing" what happens if you remove the Python and replace it with an ECL function that equally "does nothing"? Does it still fail with the same error?

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1369
Joined: Wed Oct 26, 2011 7:40 pm

Fri Sep 29, 2017 3:04 pm Change Time Zone

How big are the datasets?
richardkchapman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 108
Joined: Fri Jun 17, 2011 8:59 am

Fri Sep 29, 2017 3:13 pm Change Time Zone

In eclagent the memory limit comes from 2 places:

defaultMemoryLimitMB value in the config
#option("hthorMemoryLimit") value in the workunit

In both cases the value is specified in megabytes.
richardkchapman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 108
Joined: Fri Jun 17, 2011 8:59 am

Fri Sep 29, 2017 3:54 pm Change Time Zone

I set the following parameter in EclAgentProcess:
Code: Select all
defaultMemoryLimitMB="10000"

And, I put the following 2 statements at the top of my ECL.
Code: Select all
#option('hthorMemoryLimit',10000);
#option('outputLimitMb',10000);

Note. I had to add outputLimitMb because the default for output to the workunit is 10MB.

Now, I get this error.
Code: Select all
Error: System error: 0: Dali result outputs are restricted to a maximum of 2000 MB, the current limit is 10 MB. A huge dali result usually indicates the ECL needs altering.

Is there anything I can do to eliminate this error?
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Fri Sep 29, 2017 4:43 pm Change Time Zone

Is there a setting in the environment.xml file that would increase the Dali’s output from the current 10MB to its maximum?
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Wed Oct 11, 2017 7:46 am Change Time Zone

No. If you want larger outputs they have to be to a file.
richardkchapman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 108
Joined: Fri Jun 17, 2011 8:59 am

Mon Oct 16, 2017 8:40 pm Change Time Zone

Richard,

I started getting the dali error, "Dali result outputs are restricted to a maximum of 2000 MB", after doing what you suggested to increase EclAgentProcess memory and output to the workunit limits (see above).

I'm fairly sure that the dali error I'm getting is caused by the passing of the 2 large datasets to the embedded python routine. Why? Because I tried this 3 different ways and got the same error message: 1) the code as I have in my 1st post, above; 2) OUTPUT(COUNT(resultdataset)); and 3) OUTPUT(resultdataset,,'tlh::CMatrix',OVERWRITE).
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Tue Oct 17, 2017 8:57 am Change Time Zone

If I change the last line of your query to:

output(MatrixMultiply(A,NRowsA,NColsA,B,NRowsB,NColsB),,'result');

then I don't see any outputs to dali in the generated query. That should avoid the error.
ghalliday
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 178
Joined: Wed May 18, 2011 9:48 am

Tue Oct 17, 2017 9:22 am Change Time Zone

You also need to remove the line:

#option('outputLimitMb',10000);

If you try to set the limit higher than 2000, you will get this error, regardless of how much is actually output (in this example a couple of very small spill values are written to the workunit).
richardkchapman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 108
Joined: Fri Jun 17, 2011 8:59 am


Return to ECL

Who is online

Users browsing this forum: Bing [Bot] and 1 guest

cron