Sat Jul 02, 2022 11:20 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl ( This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.

Query publishing using remote files

Comments or questions specific to the features of ECL Watch

Wed May 10, 2017 2:23 pm Change Time Zone

Hi all,

I am trying to publish a query into a Roxie cluster but avoiding the copy of any file from Thor and instead using them remotely.

I've been reading this thread but I would like to discard other options before I dive into the configuration files:

As him, I have an index created in my Thor cluster A, I have some ECL I want to deploy in my Roxie Cluster B.
The line I use to publish is as follows:
Code: Select all
ecl publish -v --name="$folderName.$file" -s $server --target=$target -u $user -pw $password --allow-foreign --main="$folderName.$file" --memoryLimit=2GB --timeLimit=30000 --warnTimeLimit=5000 --daliip=$remoteDali --no-files --allow-foreign

A and B don't share a dali, and this works if I publish normally (Without the last 2 options). The problem is that this is a small development Roxie and it doesn't have a lot of disk space so we can't afford having Roxie to copy the files it needs.

Any Idea on how can we achieve this?

Thanks for the help!
Posts: 16
Joined: Tue May 24, 2016 4:11 pm

Wed May 10, 2017 9:05 pm Change Time Zone

This is achieved in the way your Roxie os configured.

There are two settings in the Roxie configuration that control where Roxie looks for data and index files:

copyResources Copies necessary data and key files from the current location when the query is published.

useRemoteResources Instructs Roxie to look for data and key files in the current location after the query is published.

For the scenarion you descriobe, you should set copyResources to FALSE and useRemoteResources to TRUE.

This directs the Roxie cluster to load the data from a remote location.
The query can be executed immediately, but performance is limited by network bandwidth.
This allows queries to run without using any Roxie disk space but reduces its throughput capabilities.

You can find more details about this and other Roxie configuration settings in this manual: ... df#page=10


Posts: 160
Joined: Wed May 18, 2011 1:35 pm

Thu May 11, 2017 8:41 am Change Time Zone

This was really helpful. Thanks Jim.

I will test this and thank you one more time if it works!
Posts: 16
Joined: Tue May 24, 2016 4:11 pm

Thu May 11, 2017 1:30 pm Change Time Zone

I couldn't get it to work...

This is the RoxieTopology file in the cluster, I changed those two parameters, restarted the cluster but still the files were copied over. The parameter in roxie shows now the right values but when I publish the query it copies the files again or if I specify in the publishing script the options "--no-files --allow-foreign --daliip=$remoteDali" the query gets suspended because it cannot find the file.

Any Idea what I could be missing? Platform is on version 5.6.8-1 by the way.


Posts: 16
Joined: Tue May 24, 2016 4:11 pm

Return to ECL Watch

Who is online

Users browsing this forum: No registered users and 1 guest