Sun Jul 21, 2019 8:56 am
Login Register Lost Password? Contact Us


CmdProcess and PIPE

Comments and questions related to the Enterprise Control Language

Tue Oct 30, 2012 1:42 pm Change Time Zone

While CmdProcess being discussed in another topic, I have a few questions as well.

1: it is described in the Library reference as STD.System.Util.CmdProcess(), but what is in the system is lib_fileservices.FileServices.CmdProcess() - I assume it must be the same, correct?

2: I tried an example from the reference :
OUTPUT(lib_fileservices.FileServices.CmdProcess('echo','George Jetson'));
and it just stays in the execution state, not generating any output.

3: Anyway, where the target command suppose to be executed - on each node in parallel or in one place? If this is a single place where is it? And same for PIPE.
oleg
 
Posts: 41
Joined: Fri Sep 21, 2012 9:41 am

Wed Oct 31, 2012 1:42 pm Change Time Zone

Hi Oleg,

1: it is described in the Library reference as STD.System.Util.CmdProcess(), but what is in the system is lib_fileservices.FileServices.CmdProcess() - I assume it must be the same, correct?


1. Correct. They are indeed the same, legacy version versus open source version. Depends on what cluster version you are connected to.

2: I tried an example from the reference :
OUTPUT(lib_fileservices.FileServices.CmdProcess('echo','George Jetson'));
and it just stays in the execution state, not generating any output.


2. Yes CMDProcess was reported not working in Version 3.8 but is fixed in the next update.

3: Anyway, where the target command suppose to be executed - on each node in parallel or in one place? If this is a single place where is it? And same for PIPE.


3. I think it depends on the command that you are executing. If it affects the cluster than it will be across all nodes as expected.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 984
Joined: Wed Jun 29, 2011 7:13 pm

Wed Nov 07, 2012 10:58 am Change Time Zone

bforeman wrote:2. Yes CMDProcess was reported not working in Version 3.8 but is fixed in the next update.

Is there same problem with the PIPE (action and/or OUTPUT option)?

bforeman wrote:
3: Anyway, where the target command suppose to be executed - on each node in parallel or in one place? If this is a single place where is it? And same for PIPE.


3. I think it depends on the command that you are executing. If it affects the cluster than it will be across all nodes as expected.


Hmm. It's hard to believe that ECL will know anything about UNIX command line. I.e. my understanding it should just blindly execute a process and pass arguments to it.
oleg
 
Posts: 41
Joined: Fri Sep 21, 2012 9:41 am

Wed Nov 07, 2012 12:44 pm Change Time Zone

PIPE works, at least in 3.8.4 and 3.8.6. I'm using it.

(Clarification: I'm using the PIPE version of OUTPUT; what follows talks about that, rather than the PIPE built-in function.)

Unless you go to great lengths, the command would be executed on all active nodes. Note that that doesn't mean all nodes, though. For instance, if you manipulated the data in such a way that all the records you're processing wind up on one node, then only that node's external command would be executed. In most cases, this is exactly what you want.

Where this becomes a problem is when you want to execute commands based on the overall action instead of individual records (e.g. "disable one external service before processing a group of records, then re-enable it afterwards"). You would have every node disabling and re-enabling the service in that case, so care must be taken.

Cheers,

Dan
Last edited by DSC on Wed Nov 07, 2012 3:32 pm, edited 1 time in total.
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 555
Joined: Tue Oct 18, 2011 4:45 pm

Wed Nov 07, 2012 2:18 pm Change Time Zone

Thanks, now it's getting even more interesting:

First, I've tried input pipe example from the book (BTW, It's really only example I can try because other using some non-standard commands for the pipe process):

Code: Select all
//Form 2 with XML input:
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George</FName><LName>Jetson</LName></Name>', namesRecord, XML);
OUTPUT(p);

It did output only one record. We have 50 logical nodes on 5 physical machines, so I assume this way it randomly decided which one of the nodes to use to execute this command.
Not clear, however, how it will work if the command will generate number of records - how they will move across the nodes?

But more interesting things happen after I decided to add output pipe:

Code: Select all
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George</FName><LName>Jetson</LName></Name>', namesRecord, XML);
OUTPUT(p);
OUTPUT(p,,PIPE('tee /tmp/names.all'));


Now simple output printed me 50 exactly the same rows!
So this means that OUTPUT which clearly should happen after input (I'm using the result dataset in it) is affecting the way input PIPE being executed!

Regarding the output PIPE, I've got 5 files 1 line each, i.e. 5 records total - files got overwritten since I have 10 logical nodes per one physical.
oleg
 
Posts: 41
Joined: Fri Sep 21, 2012 9:41 am

Wed Nov 07, 2012 3:27 pm Change Time Zone

This actually makes sense.

Your initial PIPE command is, in reality, creating the dataset with the echo command. It is executing on every node, so p winds up holding a single record actually located on each of your 50 nodes. The first result you see is actually the result of your first OUTPUT(), where those 50 records are created. Your second OUTPUT, with the PIPE, write the single record contained on that node to /tmp/names.all located on that node.

Edit: My earlier comments about using PIPE were explicitly about the PIPE option to OUTPUT. The PIPE built-in function is an input pipe, for getting data into the system. The PIPE() function runs on all nodes all the time, as far as I can tell.

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 555
Joined: Tue Oct 18, 2011 4:45 pm

Wed Nov 07, 2012 3:37 pm Change Time Zone

so why OUTPUT in the first sample prints just one line? I think it's a bug.
oleg
 
Posts: 41
Joined: Fri Sep 21, 2012 9:41 am

Wed Nov 07, 2012 3:45 pm Change Time Zone

I think you're right, because the following two pieces of code produce different results on Thor:

Code: Select all
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George</FName><LName>Jetson</LName></Name>', namesRecord, XML);
OUTPUT(p);


Code: Select all
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George</FName><LName>Jetson</LName></Name>', namesRecord, XML);
SEQUENTIAL(OUTPUT(p));


The second one, with the SEQUENTIAL, produces a recordset with the number of records equal to the number of nodes. If it's not a bug then something should be clarified, I think.

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 555
Joined: Tue Oct 18, 2011 4:45 pm

Wed Nov 07, 2012 8:06 pm Change Time Zone

And it becomes more interesting when you add the Node() function, like this:
Code: Select all
IMPORT STD;
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George' + STD.system.Thorlib.Node() + '</FName><LName>Jetson</LName></Name>', namesRecord, XML);
SEQUENTIAL(OUTPUT(p));
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1462
Joined: Wed Oct 26, 2011 7:40 pm

Wed Nov 07, 2012 8:06 pm Change Time Zone

And it becomes more interesting when you add the Node() function, like this:
Code: Select all
IMPORT STD;
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George' + STD.system.Thorlib.Node() + '</FName><LName>Jetson</LName></Name>', namesRecord, XML);
SEQUENTIAL(OUTPUT(p));
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1462
Joined: Wed Oct 26, 2011 7:40 pm

Next

Return to ECL

Who is online

Users browsing this forum: No registered users and 1 guest

cron