ECL ML Kmeans - SKEW error

[maniblitz]

Code: Select all

System error: 10084: Graph graph2[11], sort[19]: SORT failed. Graph graph2[11], sort[19]: Exceeded skew limit: 0.250000, estimated skew: 1.000000

After trying to cluster 800k elements on a 4 nodes cluster using kmeans, I ended up with the error listed above.

Code: Select all


IMPORT STD;
IMPORT $;
IMPORT * FROM ML;
IMPORT * FROM ML.Cluster;
IMPORT * FROM ML.Types;

### for privacy reasons, part of the code cannot be presented ###

// result is a set of 800k 300 dimensions vectors
// c is a set of 3 300 dimension vectors set as centroids

ToField(result,dresult);
ToField(c,dc);

target := KMeans(dresult,dc,30,0.3,fDist:= DF.Cosine);

output(target);



[rtaylor]

maniblitz,

SORT failed. Graph graph2[11], sort[19]: Exceeded skew limit: 0.250000, estimated skew: 1.000000

Since I don't see a SORT in your posted code, I have to assume it's in that code "### for privacy reasons, part of the code cannot be presented ###" section.

So, just addressing the error message, it's telling you the skew is 1.0 and the max skew should be 0.25 -- which tells me that all the data is being SORTed on a single node for some reason. Is it possible that every record has the same value in the SORT expression(s)?

HTH,

Richard

[maniblitz]

The content of the masked area contains two datasets that do not have any transformation operated. There is no sort, transform or distribute function involved in that part. Nevertheless, after checking the function kmeans in the Github repository, I found out that the there is a SORT function involved in the kmeans algorithm.

The code masked has no problem as it was tested before and provided the anticipated results. I which to know if there is a way to modify the skew limit for the kmeans function without having to completely modify the ECL ML source code.

[Roger Dev]

I would try making sure the records are distributed before calling Kmeans.
For example:
dresult2 := DISTRIBUTE(result, id);
dc2 := DISTRIBUTE(result, id);

Then use dresult2 and dc2 as the input to Kmeans.

If that fails, then it would be helpful if you could look into the graph (using ECLWatch) and try to identify the line of code represented by the sort (i.e. graph2, subgraph 11, activity 19).

[tlhumphrey2]

Did you randomly distribute your data across all the nodes of your cluster before executing kmeans? If not that is probably why the sort exceeded the skew limit.

[tpay]

I am currently having the same exact issue when I try to run LogisticRegression.

System error: 10084: Graph graph1[431], sort[433]: SORT failed. Graph graph1[431], sort[433]: Exceeded skew limit: 0.250000, estimated skew: 1.000000

The call to the Sort() function in ML_Core > Analysis.ecl is producing this error.

[bforeman]

Tpay,

Did you try the DISTRIBUTE as Roger suggested? If you did and you are still seeing the error, a JIRA may be needed so we can investigate further.

Regards,

Bob

[tpay]

Hi Bob,

When I use distribute ML.Analysis.Classification.Accuracy works, but ML.Analysis.Classification.AccuracyByClass still produces the same error.

Thanks
Tayfun Pay

[bforeman]

When I use distribute ML.Analysis.Classification.Accuracy works, but ML.Analysis.Classification.AccuracyByClass still produces the same error.

Thank you Tayfun! Would you please open a Jira issue on this, with sample code if possible and steps to reproduce contained in the report?
https://track.hpccsystems.com/secure/Dashboard.jspa

There is a category there for Machine Learning.

Thank You!

Bob

[tpay]

Hi Bob, Is it possible for you to take a look at this as well? viewtopic.php?uid=4453&f=10&t=8093&start=0 I can email you the Zap reports as well as well forward my email exchanges with Roger. Thanks Tayfun