Wed Jan 19, 2022 8:51 pm
Login Register Lost Password? Contact Us


Updating a superkey

Comments and questions related to the Enterprise Control Language

Thu Oct 11, 2012 2:40 pm Change Time Zone

I'm still missing something when it comes to updating a superkey that is currently in use by Roxie.

I can build the superkey and use it just fine.

When I create a new subkey and try to add it to the superkey, the process hangs until all deployed Roxie queries that use the superkey are bounced (deleted and re-added).

How can I quickly and automatically make the new data available to Roxie? Are packages the key -- ha! -- here? If so, can someone supply example code?

Thanks,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 568
Joined: Tue Oct 18, 2011 4:45 pm

Fri Oct 12, 2012 2:14 pm Change Time Zone

Dan, the last time I heard, packages support for the Community Edition would be in the next update.

Instead of having to delete and re-add the queries, did you try to simply suspend them?

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1005
Joined: Wed Jun 29, 2011 7:13 pm

Fri Oct 12, 2012 2:20 pm Change Time Zone

I did not try to bounce the query with a suspend. I'll try that next time I test an update cycle.

How would one go about triggering query management from within ECL? Execute the ecl command-line application via Std.System.Util.CmdProcess()? (Please tell me that there is another way, as that particular function is broken in 3.8.4.1CE.)

If the command-line tool is used, are you specifically talking about the deactivate/active subcommands?

Also: I believe that Package support was included in 3.6.2CE or something around there. It's the documentation that is missing.

Thanks,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 568
Joined: Tue Oct 18, 2011 4:45 pm

Fri Oct 12, 2012 2:29 pm Change Time Zone

Hi Dan,

That's unfortunate that Std.System.Util.CmdProcess() is broken, as that's the only way that I know via ECL.

Let me check with development to see if there is another way.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1005
Joined: Wed Jun 29, 2011 7:13 pm

Fri Oct 12, 2012 3:32 pm Change Time Zone

Dan,
If you change the contents of your superkey definition in dali and want to get a running roxie to use it, you can try the following command to reload everything without stopping / suspending

ecl-roxie reload


Support for packages in 3.6.x and 3.8.x is not supported (some things way work). We are working on it for 3.10 and will also be updating the documentation
sort
 
Posts: 59
Joined: Wed May 11, 2011 1:54 pm

Fri Oct 12, 2012 3:40 pm Change Time Zone

I don't seem to have a 'ecl-roxie' executable. At least, I cannot locate one within /opt/HPCCSystems. Could it be named differently?

If I did have one, though, this would be something I would have to execute on each node using Std.System.Util.CmdProcess(), right? Or is there another way to execute binaries?

Thanks,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 568
Joined: Tue Oct 18, 2011 4:45 pm

Fri Oct 12, 2012 4:04 pm Change Time Zone

sorry, the ecl-roxie command will be available in 3.10
sort
 
Posts: 59
Joined: Wed May 11, 2011 1:54 pm

Fri Oct 19, 2012 2:45 pm Change Time Zone

Since this thread is referenced elsewhere (http://hpccsystems.com/bb/viewtopic.php?f=9&t=559) I thought I'd update it with my current findings. In short, I've created a workaround that is not quite optimal (more later) but mostly works.

The core of the workaround is the following module:

Code: Select all
IMPORT * FROM Std;

EXPORT Manager := MODULE
   
   //==========================================================================
   // Record Definitions
   //==========================================================================
   
   EXPORT   RoxieQueryWorkunits := RECORD
      STRING      workUnitID;
      STRING      name;
   END;
   
   SHARED CommandRec := RECORD
      STRING      command;
   END;
   
   //==========================================================================
   // Functions
   //==========================================================================
   
   //--------------------------------------------------------------------------
   // TODO
   //--------------------------------------------------------------------------
   EXPORT   CurrentRoxieQueryWUIDList() := FUNCTION
      // Extract info on the current Roxie queries
      rs := System.Workunit.WorkunitList('',cluster:='roxie');
      
      // Pick out the few items we need
      result := PROJECT(rs,TRANSFORM(RoxieQueryWorkunits,SELF.workUnitID := LEFT.wuid,SELF.name := LEFT.job));
         
      RETURN result;
   END;
   
   //--------------------------------------------------------------------------
   // TODO
   //--------------------------------------------------------------------------
   SHARED   DoPipe(DATASET(CommandRec) rs) := FUNCTION
      doNothing := TRUE;
      
      // We need the failure handler because all the nodes will be calling
      // this function but only one will succeed
      action := OUTPUT(rs,,PIPE('/bin/bash',CSV,REPEAT)) : FAILURE(EVALUATE(doNothing));
      
      RETURN action;
   END;
   
   //--------------------------------------------------------------------------
   // TODO
   //--------------------------------------------------------------------------
   EXPORT   DisableRoxieQueries(DATASET(RoxieQueryWorkunits) rs) := FUNCTION
      // Create a record set of commands that unpublish all the queries;
      // we're not able to get the actual query name (e.g. "myquery.1") so
      // we're manually appending a '.1' to the name, and this will break if
      // there is ever a .2 or something published and active; eclwatchhost
      // is defined in /etc/hosts.
      commands := PROJECT   (
                        rs,
                        TRANSFORM   (
                                    CommandRec,
                                    SELF.command := '/usr/bin/ecl unpublish --server=eclwatchhost myroxie ' + LEFT.name + '.1'
                                 )
                     );
      
      action := DoPipe(commands);
      
      RETURN action;
   END;
   
   //--------------------------------------------------------------------------
   // TODO
   //--------------------------------------------------------------------------
   EXPORT   EnableRoxieQueries(DATASET(RoxieQueryWorkunits) rs) := FUNCTION
      // Create a record set of commands that publish all the queries;
      // eclwatchhost is defined in /etc/hosts.
      commands := PROJECT   (
                        rs,
                        TRANSFORM   (
                                    CommandRec,
                                    SELF.command := '/usr/bin/ecl publish --server=eclwatchhost --cluster=roxie --activate ' + LEFT.workUnitID
                                 )
                     );
      
      action := DoPipe(commands);
      
      RETURN action;
   END;
   
END; // Manager Module


The module is called from the following code fragment that modifies two different superkeys (slightly edited):

Code: Select all
roxieQueryInfo := Queries.Manager.CurrentRoxieQueryWUIDList() : INDEPENDENT;

actions := SEQUENTIAL   (
                     // Create the index files on disk
                     PARALLEL   (
                                 BUILD(oneIDX,DISTRIBUTED,OVERWRITE);
                                 BUILD(twoIDX,DISTRIBUTED,OVERWRITE);
                              );
                     
                     // Make sure all of our superfiles and superkeys exist
                     EnsureSuperfilesExist();
                     
                     // Disable all queries
                     Queries.Manager.DisableRoxieQueries(roxieQueryInfo);
                     
                     // Start the transaction
                     File.StartSuperFileTransaction();
                     
                     // Add new indexes to superfile
                     File.AddSuperFile(kOneSuperkey,oneIndexFileName);
                     File.AddSuperFile(kTwoSuperkey,twoIndexFileName);
                     
                     // Finalize the transaction
                     File.FinishSuperFileTransaction();
                     
                     // Enable all queries
                     Queries.Manager.EnableRoxieQueries(roxieQueryInfo);
                  );


In short, this workaround:

1) Gathers a list of all Roxie queries.

2) Creates the subkeys that will be appended to the superkeys.

3) Deletes the Roxie queries from the query set. This unlocks the superkeys for modification.

4) Adds the subkeys to the superkeys.

5) Re-adds the Roxie queries previously deleted.

The biggest problem with this implementation is that it deletes the queries. Callers would experience a hard failure while the superfile was being updated. If something besides a deletion would work here, I'd love to hear it (suspending/resuming does not work, BTW).

Other problems include 1) deleting and re-adding all queries, not just the ones associated with the affected superkeys (but that's just my implementation); and 2) the inability to determine the exact job name of the published queries (any version beyond .1 would cause a problem).

Cheers,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 568
Joined: Tue Oct 18, 2011 4:45 pm


Return to ECL

Who is online

Users browsing this forum: No registered users and 1 guest