Mon Nov 29, 2021 9:25 pm
Login Register Lost Password? Contact Us


Workunit Graph

Comments or questions specific to the features of ECL Watch

Thu Jan 24, 2013 6:55 pm Change Time Zone

Is there documentation describing how to use the workunit graph? I am specifically interesting in learning about min skew, max skew etc., and how to use this information to tune my ECL program.

In addition:

1. How can I read the graph to understand how the data is distributed and processed? For example if I perform a PROJECT on a 1 million data set that is distributed across 4 nodes, I would like to see how the PROJECT behaved and which nodes participated in the operation and to what extent.

2. What is the significance of the sub graph?

Any help is appreciated
jacksock
 
Posts: 3
Joined: Thu Jan 24, 2013 6:23 pm

Thu Jan 24, 2013 9:57 pm Change Time Zone

Hi JS,

There is a little documentation at the following link:

http://hpccsystems.com/download/docs/ecl-ide-client-tools

Regarding skew, assume that you have a three node cluster with 300 records, distributed evenly on each node. We would expect 100 records per node.

The skew percentage always identifies the maximum and minimum conditions. So in the example cluster above, a skew of +200%, -100% means:

"There is one cluster processing 200% over the nominal 100 records: 100 + 200 = 300. Another cluster is operating at -100%, so 100-100 is zero." The net result is that one cluster is doing all of the work, and the other two are idle.

So skews with small percentages (in single digits) means that your distribution of data across the nodes are even and well optimized. Indeed, sometime a skew is expected, especially near the final output where the workunit is spitting out the first 100 records from a single node.

Yes, graphs are great visual tools, used with the timings they can help to identify "hot spots" in your code where improvement might be needed. The sub-graph I think simply means that the job was split into a smaller process that contributed to the overall result.

Hope this helps,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1005
Joined: Wed Jun 29, 2011 7:13 pm


Return to ECL Watch

Who is online

Users browsing this forum: No registered users and 1 guest