Fri Oct 22, 2021 6:54 pm
Login Register Lost Password? Contact Us


Dfuplus: problem with srcIP?

Post questions specific to installation or configuration for the HPCC Systems platform

Mon Jul 26, 2021 9:21 pm Change Time Zone

Hello!

I'm trying to use dfuplus from the command line to spray (import) from CSV files.
I'm guessing I'm missing some configuration/options here because it doesn't seem to stick to the srcIP I'm passing (with some exceptions) and tries to...resolve?/do something with those IP during the process.
My setup is:
    single node,
    HPCC Systems 8.2.0-2,
    DFU Version: 7 community_8.2.0-2 (exact output of /usr/bin/dfuplus --version),
    Ubuntu 20.10,
    I run /usr/bin/dfuplus on the instance itself.


I said some exceptions because when I use localhost or 127.0.0.1, it uses 127.0.0.1 just fine:
Code: Select all
$ /usr/bin/dfuplus action=spray srcip=localhost srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=some::thing dstcluster=mythor
Checking for local Dali File Server on port 7100

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on 127.0.0.1:7100 to some::thing
Submitted WUID D20210726-211856
D20210726-211856 status: queued
Failed: No Drop Zone on '127.0.0.1' configured at '/var/lib/HPCCSystems/mydropzone/something.csv'.

Code: Select all
$ /usr/bin/dfuplus action=spray srcip=127.0.0.1 srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=some::thing dstcluster=mythor
Variable spraying from /var/lib/HPCCSystems/mydropzone/archwayha-cms/lds/2019/extracted/dme_claimsk_lds_5_2019q1.csv on 127.0.0.1:7100 to archwayha-cms::lds::2019::dme::q1
Submitted WUID D20210726-204329
D20210726-204329 status: queued
Failed: No Drop Zone on '127.0.0.1' configured at '/var/lib/HPCCSystems/mydropzone/something.csv'.


So I understand my Drop Zone is not "on" 127.0.0.1, and it expects the private IP here (please correct me if I'm wrong...and where is this defined exactly? I don't see it in my /etc/HPCCSystems/environment.xml).

But when I use my private IP I get the following:
Code: Select all
$ /usr/bin/dfuplus action=spray srcip=172.32.5.210 srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=some::thing dstcluster=mythor
Checking for local Dali File Server on port 7100

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on�V to some::thing
[20052: 2021-07-26 20:49:40 GMT: SprayVariable to some::thing: cannot resolve source network IP from ÅV.]

I'm not sure what's going on (I don't get any logs when doing this), but I wonder if it has something to do about that IP address somehow garbled here into this mess of either �V or ÅV (and it changes every time I run that command).
Any idea what I'm missing or did wrong?
Any idea how I can troubleshoot further?


By the way, I tried with my IPv6 address (just to see what would happen), but I get the same result as with localhost/127.0.0.1 (which might be normal then):
Code: Select all
$ /usr/bin/dfuplus action=spray srcip=fe80::106f:6cff:fec2:2c7f srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=some::thing dstcluster=mythor
Checking for local Dali File Server on port 7100

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on fe80::106f:6cff:fec2:2c7f to some::thing
Submitted WUID D20210726-205038
D20210726-205038 status: queued
Failed: No Drop Zone on 'fe80::106f:6cff:fec2:2c7f' configured at '/var/lib/HPCCSystems/mydropzone/something.csv'.


When I use the Spray feature in ECL Watch it works just fine, but I would like to do it from the command line (need to script the spraying of a bunch of files).

Thanks!
lpezet
 
Posts: 75
Joined: Wed Sep 10, 2014 3:14 am

Tue Jul 27, 2021 3:24 pm Change Time Zone

I might be confused between srcip and server. I think I always omitted srcip (since I always run dfuplus on the instance itself).
Regardless, I went ground zero to test things on equal footing.
I downloaded HPCCSystems 8.2.0-2 VM (VirtualBox) and HPCCSystems 7.10.70-1 VM (VirtualBox).
I simply run the VMs one at a time when performing my tests here (I run one, perform the test, shut it down, and run the other one). I am NOT changing any settings (no configmgr, no editing OS files, nada, nichts, rien).
I ran the same exact command on both VMs.
With 7.10, I get the following:
Code: Select all
$ /usr/bin/dfuplus action=spray srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=test::something dstcluster=mythor server=127.0.0.1

srcip not specified - assuming spray from local machine
Checking for local Dali File Server on port 7100

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on 192.168.56.101:7100 to test::something
Submitted WUID D20210727-110739
D20210727-110739 status: queued
D20210727-110739 Finished
Total time taken 0 secs

With 8.2:
Code: Select all
$ /usr/bin/dfuplus action=spray srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=test::something dstcluster=mythor server=127.0.0.1

srcip not specified - assuming spray from local machine
Checking for local Dali File Server on port 7100

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on http://127.0.0.1:8010/FileSpray (3) to test::something
[20052: 2021-07-27 15:21:40 GMT: SprayVariable to test::something: cannot resolve source network IP from http://127.0.0.1:8010/FileSpray (3).]

So, what am I doing wrong?

Thanks for the help!
lpezet
 
Posts: 75
Joined: Wed Sep 10, 2014 3:14 am

Tue Jul 27, 2021 3:58 pm Change Time Zone

Figured it out.
If I add srcplane=mydropzone to the dfuplus command for 8.2, it works:
Code: Select all
$ /usr/bin/dfuplus action=spray srcfile=/var/lib/HPCCSystems/mydropzone/something.csv format=csv dstname=test::something dstcluster=mythor server=127.0.0.1 srcplane=mydropzone

Variable spraying from /var/lib/HPCCSystems/mydropzone/something.csv on mydropzone to test::something
Submitted WUID D20210727-113924
0% Done
D20210727-113924 Finished
Total time taken 0 secs

I saw that new option in the dfuplus usage and on github but not in the doc (https://cdn.hpccsystems.com/releases/CE ... .2.4-1.pdf).
Had to check some tests to figure out what to set it to (https://github.com/hpcc-systems/HPCC-Pl ... t.ecl#L164).
lpezet
 
Posts: 75
Joined: Wed Sep 10, 2014 3:14 am

Fri Jul 30, 2021 6:26 pm Change Time Zone

Thanks for calling this to our attention. I will get more details and update the latest DFUPlus documentation.

I opened a Jira:
https://track.hpccsystems.com/browse/HPCC-26298

Jim
JimD
 
Posts: 157
Joined: Wed May 18, 2011 1:35 pm


Return to Installation

Who is online

Users browsing this forum: No registered users and 1 guest