Mon Nov 29, 2021 9:01 pm
Login Register Lost Password? Contact Us


Error messages meaning ?

Topics specific to the use of the ECL IDE

Tue Jun 26, 2012 9:27 pm Change Time Zone

Hello,

Is there some error messages manual in which would be explained for each error its meaning and some idea on how to handle with it ?

I have some error in a THOR cluster (AWS platform with a 3 instances thor cluster):

Error: System error: -1: Graph[21], localresultwrite[29]: SLAVE 10.91.1.230:6600: Partial row written at end of file 23 of 24 (0, 0), -1,

By investigating my code and the graph, I can see it happens in this context, where I am looking for either one STOP record at least in the dataset to empty it, or only CONT(inue) records.

Code: Select all
   INTEGER ifstatut := IF(EXISTS(listeVariables(statut=StatutVariable.STOP)),StatutVariable.VIDER,StatutVariable.CONT);
   listeVariables2 := listeVariables(statut=ifstatut);


It seems parallel filtering of listeVariables is clashing with error mentionned above because of an incomplete-row story but I don't understand much more about this.

I am investigating on this error by trying something else but I would prefer to understand what HPCC is really doing to fail, when it does not happen on HTHOR, and more generally, I think it would be a good idea to have an error index to complete the documentation.

Could you help me on this specific problem ?
Thanks,
JM.
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Wed Jun 27, 2012 4:58 pm Change Time Zone

I've found a workaround better than the original.
Per chance, the resulting dataset is produced by ITERATE. I changed the way of doing it by creating a larger collector record of the result.
My finding is by accumulating the final result inside the ITERATE operation, I don't need parallel tests on a unique dataset after which seems to be the cause of confusion. Anyway, this is not an explanation and I still don't understand the error message and why parallel filtering is not possible in this case.

JM.
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Wed Jun 27, 2012 7:58 pm Change Time Zone

JM,

Glad you found a workaround for your issue.
I still don't understand the error message and why parallel filtering is not possible in this case.
Neither do I. The code snippet you included in your first post did not provide enough context to understand what you were trying to accomplish. For any future issues that come up, please try to include as much relevant code as possible so we can see the context. That would make it more likely we'll be able to see the problem and help you faster.

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1604
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jul 02, 2012 10:32 am Change Time Zone

Hello,

I think there is a bug with GRAPH nested in ITERATE, ON THOR, which is very annoying for me as I expected to run queries to filter complex data on THOR.
My last try is on VM image but I have the same behaviour on a AWS cluster.

Don't try to find a meaning in the code I provide, as I reduced it to emphasize my problem. I want this (GRAPH nested in ITERATE) to work unless you tell me it is impossible, but in this case, I don't see it in documentation.

Here is my code :

Code: Select all
TypeNoeud := ENUM(SYMBOL,VAR,NOEUD,VIDE,LIEN);

Noeud := RECORD
   INTEGER id;         
   INTEGER idmax;   
   INTEGER symbol;
   STRING chaine;
   TypeNoeud type;
   INTEGER gauche;   
   INTEGER droite;   
END;

Btree := RECORD
   DATASET(Noeud) dn;
END;

StatutContrainte := ENUM(ACTIVE,PASSIVE,DELETED);

Contrainte := RECORD
   INTEGER id;
   Btree b;
   StatutContrainte statut;
END;

TypeModele := ENUM(FIXE,REMOVABLE);

Modele := RECORD
   INTEGER id;
   Btree b;
   TypeModele type;
END;

StatutVariable := ENUM(CONT,STOP,INUTILE,VIDER);

Variable := RECORD
   INTEGER symbol;
   Btree val;
   StatutVariable statut ;
   
END;

StatutTuple := ENUM(NONVIDE,VIDE);

TupleVariable := RECORD
   DATASET(Variable) dv;
   INTEGER p;
   INTEGER c;
   SET OF INTEGER sc;
   StatutTuple statut;
   TypeModele typeModele;
   DATASET({INTEGER id}) fusionContraintesSupprimes;
END;


StatutChoixTuple := ENUM(NONVIDE,VIDE);

ChoixTuplesVariables := RECORD
   DATASET(TupleVariable) dt;
   StatutChoixTuple statut;
END;

ListeContraintes := RECORD
   DATASET(Contrainte) lc;
   DATASET({INTEGER id}) fusionContraintesSupprimes;
END;







Noeud f9(Noeud n,INTEGER offset) := TRANSFORM
   SELF.id:=offset;
   SELF := n;
END;

DATASET(Noeud) f8(DATASET(Noeud) dn,INTEGER offset) := FUNCTION
   RETURN PROJECT(dn,f9(LEFT,offset));
END;

DATASET(Noeud) f7(DATASET(Noeud) gauche) := FUNCTION
   offset := COUNT(gauche);
   droite2 := f8(gauche,offset);
   RETURN droite2;
END;

DATASET(Noeud) f6(DATASET(Noeud) modele_cur) := FUNCTION
   RETURN f7(modele_cur);
END;


DATASET(Noeud) f5(SET OF DATASET(Noeud) modele,INTEGER d) := FUNCTION
   modele_cur := modele[d-1];
   RETURN    f6(modele_cur);
END;



DATASET(Noeud) f4(Btree modele) := FUNCTION
   listeNoeudsContrainte := GRAPH(modele.dn,COUNT(modele.dn),f5(ROWSET(LEFT),COUNTER));
   RETURN listeNoeudsContrainte;
END;


Variable f3(Variable modele,Variable var,INTEGER c) := TRANSFORM
//   SELF.val.dn := f4(IF(TRUE,var.val,modele.val)); // this works with var.val !
//   SELF.val.dn := f4(IF(c=10000,var.val,modele.val)); // this bug because of modele.val !
   SELF.val.dn := f4(IF(c=1,var.val,modele.val)); // this bug because of counter and modele.val !?!
   SELF := var;
END;







listeTuplesLc := DATASET([{DATASET([{2844991738,{DATASET([{1,1,703514648,'a',1,-1,-1}],Noeud)},1}],Variable),1,1,[],1,1,[]}],TupleVariable);
listeChoixTuples := DATASET([{listeTuplesLc,1}],ChoixTuplesVariables);
lChoixTuples := listeChoixTuples[1];
TupleVariable tuple := lChoixTuples.dt[1];

DATASET(Variable) lVar := ITERATE(tuple.dv,f3(LEFT,RIGHT,COUNTER));

OUTPUT(lVar);



Simply, you will find commented in function f3 different alternatives : one works and others don't on THOR but all work on HTHOR.

I would be very nice if you could help me on this. And if there is some fix available, how to install sources with the fix inside.
Again, "error manual" is cruelly missing here. Debugger does not work in this case, I don't konw why.

Thanks,
JM.
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Mon Jul 02, 2012 1:34 pm Change Time Zone

JM,
Simply, you will find commented in function f3 different alternatives : one works and others don't on THOR but all work on HTHOR.
Before the Open Source releases, both the LOOP and GRAPH functions were documented to only operate on Roxie and hThor (not Thor). With the move to Open Source, both these functions are now meant to work on Thor too. Obviously, GRAPH is not. I will report the issue.
I would be very nice if you could help me on this. And if there is some fix available, how to install sources with the fix inside.
Help to/from the HPCC user community is precisely what this forum is about. Your previous questions have been answered, as will your future questions. At some point you will be well enough informed to begin helping others with their issues. That's what Community is all about.
Debugger does not work in this case, I don't konw why.
The Debugger only works on hThor and Roxie, because these both have a single node that the code runs on. Thor, however, has code running on multiple nodes simultaneously, making debugging extremely problematic.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1604
Joined: Wed Oct 26, 2011 7:40 pm

Mon Jul 02, 2012 10:19 pm Change Time Zone

Richard,

Thank you for your answer. I've lost one week trying to fix this issue (no debugger and code pretty complex). Everything is pretty clear now and I re-made GRAPH actions in something else with, I must confess, some benefit to code optimization.

After fixing GRAPH's code, I found also that :
Error: System error: 0: Graph[310], Global DEDUP,ALL is not supported
Error: System error: 0: Graph[21], Thor currently, does not support a dataset loop condition, activity id: 23
The documentation doesn't say anything on this. I am trying now to find an alternative to LOOP at least.

I am not sure to understand perfectly the exact difference between THOR and ROXIE.
I tested some simple code on a THOR AWS cluster and saw it is scalable, gaining performances by adding servers (this is was I am looking for). This is not the case with (multi-node) ROXIE, as only one server was loaded during query run. I did not found much information on the web and my current knowledge is that Roxie is just adding some indexes to accelerate queries on Thor (and may eventually deals with multi user connexions). But performance tests show me the contrary.
Now, can you confirm or invalidate that THOR is the only platform (vs Roxie) that allow scalability in a distributed multi server architecture ?

Thanks,
JM.
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Tue Jul 03, 2012 12:47 pm Change Time Zone

JM,
I am not sure to understand perfectly the exact difference between THOR and ROXIE.
I tested some simple code on a THOR AWS cluster and saw it is scalable, gaining performances by adding servers (this is was I am looking for). This is not the case with (multi-node) ROXIE, as only one server was loaded during query run. I did not found much information on the web and my current knowledge is that Roxie is just adding some indexes to accelerate queries on Thor (and may eventually deals with multi user connexions). But performance tests show me the contrary.
Now, can you confirm or invalidate that THOR is the only platform (vs Roxie) that allow scalability in a distributed multi server architecture ?
This white paper (http://cdn.hpccsystems.com/whitepapers/wp_introduction_HPCC.pdf) goes into great detail about the HPCC platform and the differences between Thor and Roxie, but here's the brief description:

Thor is a massively parallel cluster designed to do ETL (Extract, Transform, and Load) work -- the kind of data ingest and transformation jobs that every big data shop needs to do to take raw data and transform it into useful information that can either be sold to external customers or used by internal customers. Data is distributed across the nodes and the same "work" is done in parallel by each node on the data that that node is given to work on. IOW, we distribute the work by how we distribute the data. Thor is a "back office" tool (running just one job at a time) that builds data products from raw, un-indexed data, and creates whatever the "saleable" format needs to be (datasets and/or indexes).

Roxie is a customer interface tool designed to serve up data requests to end-users, handling thousands of concurrent requests at a time. Unlike Thor, a single Roxie query is mostly (with few exceptions) handled by a single Roxie node, with support from whichever nodes in the cluster contains the data needed for each particular query. Most Roxie queries use indexes for fast performance. Roxie is a "front counter" tool that serves up information to customers.

Thor is used to create the data that Roxie will use. ETL and query development work are all done on Thor. A query (and its required data) is only published to Roxie when the final data/query combination is complete. Once published to Roxie, the query is available for end-users to use.

So, as an ECL developer, 99%+ of all your work is done on Thor -- Roxie receives the end result of your work and allows customers to reap the benefits of all your work.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1604
Joined: Wed Oct 26, 2011 7:40 pm

Thu Jul 05, 2012 9:10 am Change Time Zone

ideal wrote:I think there is a bug with GRAPH nested in ITERATE, ON THOR, which is very annoying for me as I expected to run queries to filter complex data on THOR.
My last try is on VM image but I have the same behaviour on a AWS cluster.

Don't try to find a meaning in the code I provide, as I reduced it to emphasize my problem. I want this (GRAPH nested in ITERATE) to work unless you tell me it is impossible, but in this case, I don't see it in documentation.


ideal wrote:Simply, you will find commented in function f3 different alternatives : one works and others don't on THOR but all work on HTHOR.


Hi,

are you using build 3.6.2 ? If so, there was a bug affecting GRAPH and LOOP there, that caused some child queries to spuriously throw an error when they were run : "Global child graph? : Global acts = ..".
I've run your example and reproduced that bug in 3.6.2

In 3.8 those issues are fixed, are you able to try the latest 3.8 release candidate that's available on the portal?
jsmith
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 81
Joined: Tue Jul 19, 2011 12:58 pm

Thu Jul 05, 2012 9:44 pm Change Time Zone

Hello,

It solved the problem, at least on this sample above.

Thanks,
JM
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Fri Jul 06, 2012 3:00 pm Change Time Zone

After changing my code to avoid above issues, I come back now to the loop command, hoping it can work with this new release as it would be very convenient to have it.
Apparently we still have the same problem :

Code: Select all
Error:    System error: 0: Graph[1], Thor currently, does not support a dataset loop condition, activity id: 7 (0, 0), 0,


I checked the hpcc system version by looking on EclWatch and get this :
Enterprise Services Platform
community_3.8.0-4rc

At first try, there was a compiler mismatch but still the same loop error after configuring the right compiler.

Then, is it possible to do a real loop in ecl (not just simulated with iterate) ?
JM.
ideal
 
Posts: 86
Joined: Tue Jun 12, 2012 1:17 pm

Next

Return to ECL IDE

Who is online

Users browsing this forum: No registered users and 1 guest