Tue Aug 21, 2018 11:56 pm
Login Register Lost Password? Contact Us


Dali result outputs are restricted to a maximum of 2000 MB

Comments and questions related to the Enterprise Control Language

Fri Oct 13, 2017 7:07 pm Change Time Zone

Below is my ecl code. Here is the full error message I'm getting.
Code: Select all
Error: System error: 0: Dali result outputs are restricted to a maximum of 2000 MB, the current limit is 10 MB. A huge dali result usually indicates the ECL needs altering.


The 2 datasets, A3 and B3, are large. Basically these are 2 matrices. The dimensions of the first one is 11410 x 40000 and the second one's dimensions are 40000 x 11310. I'm calling STD.BLAS.dgemm to do a matrix multiply.

Any one know how I can get around this error?

Code: Select all
#option('hthorMemoryLimit',10000);
#option('outputLimitMb',10000);
IMPORT STD;
IMPORT PBblas;
IMPORT PBblas.Types as Types;
Layout_Cell := Types.Layout_Cell;

A3 := DATASET('~hthor::tlh::AMatrix_PBblas',Layout_Cell,THOR);
B3 := DATASET('~hthor::tlh::BMatrix_PBblas',Layout_Cell,THOR);

A3_Set := SET(A3, v);
B3_Set := SET(B3, v);
C := STD.BLAS.dgemm(FALSE, FALSE, 11410, 11410, 40000, 1.0, A3_Set, B3_Set);
OUTPUT(COUNT(C));
//OUTPUT(DATASET([C],{REAL8 v}),,'tlh::CMatrix_PBblas',OVERWRITE);
tlhumphrey2
 
Posts: 251
Joined: Mon May 07, 2012 6:23 pm

Fri Oct 13, 2017 7:58 pm Change Time Zone

I forgot to mention that I replaced the OUTPUT(COUNT with the OUTPUT to a file that I have commented out and got the same error message.
tlhumphrey2
 
Posts: 251
Joined: Mon May 07, 2012 6:23 pm

Mon Oct 16, 2017 12:11 pm Change Time Zone

I would try a group rollup on each of the two datasets creating a singe record per dataset of {SET OF REAL8 mat}

Given the 2 new record sets, I would use COMBINE to bring the two records together in the same transform, and then call the BLAS routine in the transform for the COMBINE(...) function.
john holt
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 22
Joined: Mon Jun 25, 2012 12:43 pm

Tue Oct 17, 2017 4:48 pm Change Time Zone

John,

Will the following do the same thing as the group rollup followed by the combine?

Code: Select all
#option('hthorMemoryLimit',10000);
#option('outputLimitMb',10000);
IMPORT STD;
IMPORT PBblas;
IMPORT PBblas.Types as Types;
Layout_Cell := Types.Layout_Cell;

A := DATASET('~hthor::tlh::AMatrix_PBblas',Layout_Cell,THOR);
B := DATASET('~hthor::tlh::BMatrix_PBblas',Layout_Cell,THOR);

SET OF REAL8 A_Set := SET(SORT(A,y,x), v);
SET OF REAL8 B_Set := SET(SORT(B,y,x), v);

aMatrixRec := RECORD
  SET OF REAL8 mat;
END;

A_DS := DATASET([{A_Set}], aMatrixRec);
B_DS := DATASET([{B_Set}], aMatrixRec);

aMatrixRec matrixMultiply(aMatrixRec A, aMatrixRec B) := TRANSFORM
  SELF.mat := STD.BLAS.dgemm(FALSE, FALSE, 11410, 11410, 40000, 1.0, A.mat, B.mat);
END;

C_DS := COMBINE(A_DS, B_DS,matrixMultiply(LEFT,RIGHT));

//OUTPUT(COUNT(C_DS));
OUTPUT(C_DS,,'tlh::CMatrix_PBblas',OVERWRITE);


Tim
tlhumphrey2
 
Posts: 251
Joined: Mon May 07, 2012 6:23 pm

Tue Oct 17, 2017 4:53 pm Change Time Zone

You need to get rid of the line

#option('outputLimitMb',10000);

Or nothing will work
richardkchapman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 108
Joined: Fri Jun 17, 2011 8:59 am

Tue Oct 17, 2017 5:22 pm Change Time Zone

Tim,
You still have the huge SET OF REAL8 objects. These need to be removed.

Start with your A and B datasets and do the rollups, BUT, only if all of the positions are present as records. If you are missing any records, you will need to use the PBblas function. The PBblas function does not need to sort the records before constructing the array.
john holt
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 22
Joined: Mon Jun 25, 2012 12:43 pm


Return to ECL

Who is online

Users browsing this forum: No registered users and 1 guest