Query publishing using remote files
Hi all,
I am trying to publish a query into a Roxie cluster but avoiding the copy of any file from Thor and instead using them remotely.
I've been reading this thread but I would like to discard other options before I dive into the configuration files:
viewtopic.php?f=17&t=1992
As him, I have an index created in my Thor cluster A, I have some ECL I want to deploy in my Roxie Cluster B.
The line I use to publish is as follows:
A and B don't share a dali, and this works if I publish normally (Without the last 2 options). The problem is that this is a small development Roxie and it doesn't have a lot of disk space so we can't afford having Roxie to copy the files it needs.
Any Idea on how can we achieve this?
Thanks for the help!
Daniel.
I am trying to publish a query into a Roxie cluster but avoiding the copy of any file from Thor and instead using them remotely.
I've been reading this thread but I would like to discard other options before I dive into the configuration files:
viewtopic.php?f=17&t=1992
As him, I have an index created in my Thor cluster A, I have some ECL I want to deploy in my Roxie Cluster B.
The line I use to publish is as follows:
- Code: Select all
ecl publish -v --name="$folderName.$file" -s $server --target=$target -u $user -pw $password --allow-foreign --main="$folderName.$file" --memoryLimit=2GB --timeLimit=30000 --warnTimeLimit=5000 --daliip=$remoteDali --no-files --allow-foreign
A and B don't share a dali, and this works if I publish normally (Without the last 2 options). The problem is that this is a small development Roxie and it doesn't have a lot of disk space so we can't afford having Roxie to copy the files it needs.
Any Idea on how can we achieve this?
Thanks for the help!
Daniel.
- dsanchez
- Posts: 16
- Joined: Tue May 24, 2016 4:11 pm
This is achieved in the way your Roxie os configured.
There are two settings in the Roxie configuration that control where Roxie looks for data and index files:
copyResources Copies necessary data and key files from the current location when the query is published.
useRemoteResources Instructs Roxie to look for data and key files in the current location after the query is published.
For the scenarion you descriobe, you should set copyResources to FALSE and useRemoteResources to TRUE.
This directs the Roxie cluster to load the data from a remote location.
The query can be executed immediately, but performance is limited by network bandwidth.
This allows queries to run without using any Roxie disk space but reduces its throughput capabilities.
You can find more details about this and other Roxie configuration settings in this manual:
http://cdn.hpccsystems.com/releases/CE- ... df#page=10
HTH,
Jim
There are two settings in the Roxie configuration that control where Roxie looks for data and index files:
copyResources Copies necessary data and key files from the current location when the query is published.
useRemoteResources Instructs Roxie to look for data and key files in the current location after the query is published.
For the scenarion you descriobe, you should set copyResources to FALSE and useRemoteResources to TRUE.
This directs the Roxie cluster to load the data from a remote location.
The query can be executed immediately, but performance is limited by network bandwidth.
This allows queries to run without using any Roxie disk space but reduces its throughput capabilities.
You can find more details about this and other Roxie configuration settings in this manual:
http://cdn.hpccsystems.com/releases/CE- ... df#page=10
HTH,
Jim
- JimD
- Posts: 160
- Joined: Wed May 18, 2011 1:35 pm
This was really helpful. Thanks Jim.
I will test this and thank you one more time if it works!
I will test this and thank you one more time if it works!
- dsanchez
- Posts: 16
- Joined: Tue May 24, 2016 4:11 pm
I couldn't get it to work...
This is the RoxieTopology file in the cluster, I changed those two parameters, restarted the cluster but still the files were copied over. The parameter in roxie shows now the right values but when I publish the query it copies the files again or if I specify in the publishing script the options "--no-files --allow-foreign --daliip=$remoteDali" the query gets suspended because it cannot find the file.
Any Idea what I could be missing? Platform is on version 5.6.8-1 by the way.
Thanks!
EDIT*
This is the RoxieTopology file in the cluster, I changed those two parameters, restarted the cluster but still the files were copied over. The parameter in roxie shows now the right values but when I publish the query it copies the files again or if I specify in the publishing script the options "--no-files --allow-foreign --daliip=$remoteDali" the query gets suspended because it cannot find the file.
Any Idea what I could be missing? Platform is on version 5.6.8-1 by the way.
Thanks!
EDIT*
- dsanchez
- Posts: 16
- Joined: Tue May 24, 2016 4:11 pm
4 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 1 guest