Mon Jul 04, 2022 6:58 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl ( This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.

Memory Pool Exhausted error when passing 2 large datasets

Post questions specific to installation or configuration for the HPCC Systems platform

Fri Sep 29, 2017 2:51 pm Change Time Zone

Below is the code. Currently, my embedded python doesn't do anything. The ECL reads in 2 large datasets and passes both to the embedded python, MatrixMultiply. The sizes of the 2 datasets are 1,600,040,000 and 1,600,080,000. I get the following error:
Code: Select all
Error: System error: 1301: Memory pool exhausted: pool id 4194314 (1216 pages) exhausted, requested 1 (in Disk Read G22 E23)

Code: Select all
import python;

rec0 := RECORD
  REAL cell;

rec := RECORD
DATASET(rec0) arow;

DATASET(rec) MatrixMultiply(DATASET(rec) A, unsigned nrowsA, unsigned ncolsA,DATASET(rec) B, unsigned nrowsB, unsigned ncolsB) := embed(Python)
    import numpy as np
    import re
    return A




I'm running this on hthor. I'm working on a machine that has 15GB of memory. So, I should have plenty of memory. Both datasets should fit entirely in memory. I want to make changes to my environment.xml file so more memory will be available to my workunit. I have added these 2 parameters to both EclAgentProcess and ThorCluster. But, I'm still getting the error.
Code: Select all

Ang help would be most appreciated.
Posts: 260
Joined: Mon May 07, 2012 6:23 pm

Fri Mar 16, 2018 1:32 pm Change Time Zone

Hi Tim,

This might be something to post to the JIRA issue tracker, I just saw a similar post this morning with a developer attempting a Python script with PIPE. Could be related:

Community Advisory Board Member
Community Advisory Board Member
Posts: 1006
Joined: Wed Jun 29, 2011 7:13 pm

Return to Installation

Who is online

Users browsing this forum: No registered users and 1 guest