Sun Mar 18, 2018 11:43 am
Login Register Lost Password? Contact Us

Parallel Embed - R, Python, etc..

Questions around writing code and queries

Thu Feb 08, 2018 6:35 pm Change Time Zone

Is there a way, and how would one go about coding, to have an embed structure parallelized, assuming the plugins are on each and every node? Specifically, I am interested in have python embeds to be parallelized. i.e. have my python code run separately on each node. ... ntegration ... cture.html

I know PIPE can be used to parallelize some tasks, but I was hoping to use embed. I have tried using LOCAL on the dataset that is being passed into my embed to no avail.

Thanks in advance
Posts: 2
Joined: Thu Feb 08, 2018 6:30 pm

Thu Feb 08, 2018 7:42 pm Change Time Zone


Since every Thor node runs exactly the same .so file, and each node simply operates on whatever data is on that node (for operations that don't need to swap data between the nodes for correct execution), I would assume that your embedded Python code would do the same. IOW, parallel operation is the default mode on a multi-node Thor.

What happens when you try running a test job?


Community Advisory Board Member
Community Advisory Board Member
Posts: 1337
Joined: Wed Oct 26, 2011 7:40 pm

Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest