Publishing queries with remote dali

[alex]

I was hoping I would be able to keep the code agnostic about where the file lives and just use the "remote dali" feature of published queries.

I know using foreign::<remote dali ip>::path::to::file works, as I've tested it (so the file is definitely where I think it is, and authorization doesn't seem to be an issue). I'm specifically trying avoid having to do that, though.

[rtaylor]

Alex,

OK, I just tested this. Here are the steps I took:

• I built an INDEX on my Training1 cluster with a local name for the file (so it only exists on my Training1 Thor).
• Then I wrote a Roxie query for that INDEX on my Training2 cluster (no, they do not share the same Dali).
• On Training2, I compiled the query and Published it to my Training2 Roxie. I specified the Training1 Dali IP as the Remote Dali.
• It published with no errors and ran with no errors.

Therefore, it is not necessary to use "Foreign file" naming (I didn't) to accomplish this.

So, next you'll want to duplicate my experiment with your code and data, step-by-step, and see if you still get an error.

HTH,

Richard

[alex]

The only difference I see between what you're doing and what I'm doing is that in my simple experiment, my file is thor file and not an index. Other than that my steps are exactly the same as yours.

I presume that shouldn't make a difference, though.

[rtaylor]

Alex,

Can you upload all your test code so we can look at it, please?

Richard

[alex]

This is hand-typed so there might be typos; My HPCCs are not on an Internet-connected network.

Code: Select all

string3 codein1 := '' : stored('ISO3');
string40 namein1 := '' : stored('Name');

countries := DATASET('~120way::Country_To_ISO3', Layout_Country_to_ISO3, thor);
d1 := countries(
if( codein1 != '', (trim(codein1) = iso3_code), true),
if( namein1 != '', stringlib.stringfind(trim(country), trim(namein1), 1) >0, true));

output(d1, named('Matching_Countries'));



[bforeman]

Also, what we would really like to see is the exact content of the error. It would usually have a logical file name in it.

The reason its failing could be for so many reasons. Like incorrect use of ~ or other naming issues. Or perhaps it can’t get to the dali, but the error could help us diagnose.

Bob

[alex]

Easy enough:

State: suspended
Error: Could not resolve filename 120way::Country_To_ISO3

I've fiddled with tildes; the error is the same if I leave it out of the code, but filename is obviously wrong since its appends the cluster name to the front of the path.

ETA: As I mentioned, if I use "foreign::<ip address>" in front of the path in the DATASET, it resolves the file ok. So it's not a path issue in that respect.

[rtaylor]

Alex,

I see a couple of possible things:

• "120way" appears to be the scope name of your cluster, and you have no directory specified (since directories must follow definition naming rules, which do not allow for name starting with a number), so I would suggest trying with a file that is in a directory under your scope
• Roxie queries are most often contained within FUNCTION structures (really old ones may still use MACRO) and you code does not, so you may want to change that

Here's my query code:

Code: Select all

IMPORT $;
EXPORT TestPublishRemoteDaliService() := FUNCTION

  STRING30 fname_value := '' : STORED('FirstName');
  STRING30 lname_value := '' : STORED('LastName');

  Fetched := IF(fname_value <> '',
                $.File_People.IDX_LFname(LastName=  lname_value,
                                         FirstName=  fname_value),
                $.File_People.IDX_LFname(LastName=  lname_value));

  RETURN Fetched;
END;

And this is the supporting code:

Code: Select all

EXPORT File_People := MODULE
  Layout := RECORD
    UNSIGNED8 ID;
    STRING15  FirstName;
    STRING25  LastName;
    STRING15  MiddleName;
    STRING2   NameSuffix;
    STRING8   FileDate;
    UNSIGNED2 BureauCode;
    STRING1   MaritalStatus;
    STRING1   Gender;
    UNSIGNED1 DependentCount;
    STRING8   BirthDate;
    STRING42  StreetAddress;
    STRING20  City;
    STRING2   State;
    STRING5   ZipCode;
  END;
  EXPORT File := DATASET('~CLASS::RT::Intro::Persons',Layout,FLAT);
  EXPORT IDX_LFname  := INDEX(File,{Lastname,Firstname},{ID}, 
                              '~RTTEST::RemoteTest::KEY::People::LFname');
  EXPORT BLD_IDX_LFname  := BUILD(IDX_LFname,OVERWRITE);
END;

HTH,

Richard

[alex]

In response to your 2 points:

1) Names have been sanitized for public viewing; I typed in "120way" but that's not the actual name of the cluster. The actual path in my code is correct; publishing the query to the cluster that actually contains the file works ok. It's only when publishing it on the remote cluster that the file fails to resolve.

2) Wrapping the code in a function or macro has no effect on the ability to resolve the file.

[bforeman]

OK, one last request please, can you post the Roxie log that contains the error? Since the error is reflected on every node, any log from any node will do, as long as it contains the error.

Thanks,

Bob