Thu Aug 16, 2018 6:54 am
Login Register Lost Password? Contact Us


Memory Pool Exhausted error when passing 2 large datasets

Post questions specific to installation or configuration for the HPCC Systems platform

Fri Sep 29, 2017 2:51 pm Change Time Zone

Below is the code. Currently, my embedded python doesn't do anything. The ECL reads in 2 large datasets and passes both to the embedded python, MatrixMultiply. The sizes of the 2 datasets are 1,600,040,000 and 1,600,080,000. I get the following error:
Code: Select all
Error: System error: 1301: Memory pool exhausted: pool id 4194314 (1216 pages) exhausted, requested 1 (in Disk Read G22 E23)

Code: Select all
import python;

rec0 := RECORD
  REAL cell;
END;

rec := RECORD
DATASET(rec0) arow;
END;

DATASET(rec) MatrixMultiply(DATASET(rec) A, unsigned nrowsA, unsigned ncolsA,DATASET(rec) B, unsigned nrowsB, unsigned ncolsB) := embed(Python)
    import numpy as np
    import re
    return A

endembed;

A:=DATASET('~hthor::tlh::AMatrix',rec,THOR);
B:=DATASET('~hthor::tlh::BMatrix',rec,THOR);
NRowsA:=COUNT(A);
NColsA:=COUNT(A[1].arow);
NRowsB:=COUNT(B);
NColsB:=COUNT(B[1].arow);

MatrixMultiply(A,NRowsA,NColsA,B,NRowsB,NColsB);

I'm running this on hthor. I'm working on a machine that has 15GB of memory. So, I should have plenty of memory. Both datasets should fit entirely in memory. I want to make changes to my environment.xml file so more memory will be available to my workunit. I have added these 2 parameters to both EclAgentProcess and ThorCluster. But, I'm still getting the error.
Code: Select all
defaultMemoryLimit="10000000000"
totalMemoryLimit="10000000000"

Ang help would be most appreciated.
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Fri Mar 16, 2018 1:32 pm Change Time Zone

Hi Tim,

This might be something to post to the JIRA issue tracker, I just saw a similar post this morning with a developer attempting a Python script with PIPE. Could be related:

https://hpccsystems.com/bb/viewtopic.php?f=8&t=5393

Regards,
Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 975
Joined: Wed Jun 29, 2011 7:13 pm


Return to Installation

Who is online

Users browsing this forum: No registered users and 1 guest