Thu Sep 20, 2018 4:20 pm
Login Register Lost Password? Contact Us


Dfuplus fails to spray files

Topics related to recommendations or questions on the design for HPCC Systems clusters

Mon Apr 23, 2018 3:37 pm Change Time Zone

I have been trying to spray data using dfuplus command.

Here are the details of my HPCC setup.

Master IP Address :172.31.45.14
Slave 1 IP Address : 172.31.33.152
Slave 2 IP Address : 172.31.42.187

On my master node, I have 30 GB of kmeans related dataset in csv format spread across 30 files.
The directory is : /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/

Using the dfuplus command, I am trying to spray the same data across my slave nodes.

Here is the command I am running for an individual file.

$ sudo dfuplus action=spray srcip=172.31.45.14 srcfile=/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 dstname=kmeans::dataset::file1 dstcluster=mythor server=http://172.31.45.14:8010 format=csv

Checking for local Dali File Server on port 7100

Variable spraying from /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 on 172.31.45.14:7100 to kmeans::dataset::file1
Submitted WUID D20180423-152513
D20180423-152513 status: queued
Failed: No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.


The job fails stating that
No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.

My ECL Watch console shows a single mydropzone entry of 172.31.33.152.
rsghatpa
 
Posts: 13
Joined: Wed Mar 28, 2018 5:13 pm

Mon Apr 23, 2018 4:56 pm Change Time Zone

rsghatpa,

You can only spray from locations that are "registered drop zones" in your environment.

You can either move the file(s) to your drop zone or reconfigure your environment so that
mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/ on 172.31.45.14 is a registered drop zone.

HTH,

Jim
JimD
 
Posts: 134
Joined: Wed May 18, 2011 1:35 pm

Mon Apr 23, 2018 5:43 pm Change Time Zone

First, take the port number off the server. Second, use the IP addresses found on your ecl watch under the Operations icon then click on System Servers.
tlhumphrey2
 
Posts: 255
Joined: Mon May 07, 2012 6:23 pm

Mon Apr 23, 2018 6:00 pm Change Time Zone

From a linux (actually it was my master instance) I placed a file in my home directory, /home/ubuntu, and was able to spray it with the following:
Code: Select all
sudo dfuplus action=spray srcip=10.106.73.212 \
srcfile=/home/ubuntu/myfile_head4.csv \
dstname=tlh::myfile_head4 dstcluster=mythor \
server=http://10.106.73.212 format=csv
tlhumphrey2
 
Posts: 255
Joined: Mon May 07, 2012 6:23 pm

Tue Apr 24, 2018 3:03 pm Change Time Zone

rsghatpa,

Let us know if you got your spray to work and how, please.

Tim
tlhumphrey2
 
Posts: 255
Joined: Mon May 07, 2012 6:23 pm

Mon Apr 30, 2018 2:22 pm Change Time Zone

Hi Tim, it worked. I changed my dropzone location. Added my files to it. And using ECL Watch did the necessary spray jobs.
rsghatpa
 
Posts: 13
Joined: Wed Mar 28, 2018 5:13 pm

Mon Apr 30, 2018 3:01 pm Change Time Zone

rsghatpa,

Glad to hear it worked. Did you basically do the dfuplus command as I show it above?

Tim
tlhumphrey2
 
Posts: 255
Joined: Mon May 07, 2012 6:23 pm


Return to Clustering

Who is online

Users browsing this forum: No registered users and 0 guests

cron