Thu Oct 28, 2021 2:04 pm
Login Register Lost Password? Contact Us


EMBED Python execution

Questions around writing code and queries

Wed Jan 18, 2017 8:03 pm Change Time Zone

Hi,

I am working with the embed function for python in ECL. I see when I use the function it is executing on the thor master instead of the thor slaves. Is this behavior expected? Do all the embed functions work like this?

Right now I have thor master and slave on different nodes. Currently using one slave.
iMikePayne
 
Posts: 31
Joined: Wed Jul 16, 2014 12:33 am

Wed Jan 18, 2017 8:39 pm Change Time Zone

If you are executing on thor, the embed python should execute on all thor slaves (and maybe the master -- I'm not sure about the master). If you are running on hthor, it will run ONLY on the master.
tlhumphrey2
 
Posts: 260
Joined: Mon May 07, 2012 6:23 pm

Thu Jan 19, 2017 1:08 pm Change Time Zone

I am executing on thor and it is running on the master and not the slave.

Code: Select all
IMPORT Python;

namerec := RECORD
   string name;
END;

names := DATASET([{'Rec1:'}, {'Rec2:'}, {'Rec3'},{'Rec4:'}], namerec,distributed);

string getips(dataset(namerec) input) := EMBED(Python) 
import socket;
 
s = ''
for n in input:
  s = s+n.name + ' ' + socket.gethostname()+'|'
return s;
ENDEMBED;

output(getips(names));


All return the same ip address on a four node cluster.
iMikePayne
 
Posts: 31
Joined: Wed Jul 16, 2014 12:33 am

Thu Jan 19, 2017 2:22 pm Change Time Zone

I've created a JIRA for anyone interested in following:

https://track.hpccsystems.com/browse/HPCC-16907
iMikePayne
 
Posts: 31
Joined: Wed Jul 16, 2014 12:33 am

Thu Jan 19, 2017 2:27 pm Change Time Zone

Your are working on a 4 node thor. Do you know if the 4 nodes are on the same computer and therefore have the same IP address? (note: you can build a thor cluster which has more than one node per computer (IP)).
tlhumphrey2
 
Posts: 260
Joined: Mon May 07, 2012 6:23 pm

Thu Jan 19, 2017 2:48 pm Change Time Zone

Not sure as it is an internal cluster. What prompted the 4 node test is was that the AWS instance where master and slave are on separate nodes behaved the same way.
iMikePayne
 
Posts: 31
Joined: Wed Jul 16, 2014 12:33 am

Thu Jan 19, 2017 2:59 pm Change Time Zone

You can see thor slave nodes' IPs from ecl watch: Operations -> Cluster Processes -> mythor
tlhumphrey2
 
Posts: 260
Joined: Mon May 07, 2012 6:23 pm

Thu Jan 19, 2017 3:00 pm Change Time Zone

The ips are different. Ending in .1-.5. 1 being master.
iMikePayne
 
Posts: 31
Joined: Wed Jul 16, 2014 12:33 am


Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest