Skip to main content

External Service Implementation

ECL external system services are implemented as exported functions in a .SO (Shared Object). An ECL system service .SO can contain one or more services and (possibly) a single .SO initialization routine. All system service libraries must be thread safe.

All exported functions in the .SO (hereafter referred to as "entry points") must adhere to certain calling and naming conventions. First, entry points must use the "C" naming convention. That is, function name decoration (like that used by C++) is not allowed.

Second, the storage class of __declspec(dllexport) and declaration type _cdecl need to be declared for Windows/Microsoft C++ applications. Typically, SERVICE_CALL is defined as _declspec(dllexport) and SERVICE_API is defined as _cdecl for Windows, and left as nulls for Linux. For example:

Extern "C" _declspec(dllexport) unsigned _cdecl Countchars(const unsigned len, const char *string)

Note: The use of an external SERVICE may be restricted to signed modules. See Code Signing in the ECL Programmer's Guide.

.SO Initialization

The following is an example prototype for an ECL (.SO) system service initialization routine:

extern "C" void stdcall <functionName> (IEclWorkUnit *w);

The IEclWorkUnit is transparent to the application, and can be declared as Struct IEclWorkUnit; or simply referred to as a void *.

In addition, an initialization routine should retain a reference to its "Work Unit." Typically, a global variable is used to retain this value. For example:

IEclWorkUnit *workUnit;
     // global variable to hold the Work Unit reference
  
  extern "C" void SERVICE_API myInitFunction (IEclWorkUnit *w)
  {
       workUnit = w; // retain reference to "Work Unit"
  }

Entry Points

Entry points have the same definition requirements as initialization routines. However, unlike initialization routines, entry points can return a value. Valid return types are listed below. The following is an example of an entry point:

extern "C" __int64 SERVICE_API PrnLog(unsigned long len, const char *val)
  {
  }

SERVICE Structure - external

For each system service defined, a corresponding ECL function prototype must be declared (see SERVICE Structure).

  servicename := SERVICE
    functionname(parameter list) [: keyword = value];
    END;
  
  For example:
  email := SERVICE
    simpleSend(STRING address, STRING template, STRING subject)
       : LIBRARY='ecl2cw', INITFUNCTION='initEcl2Cw';
     END;

Keywords

This is the list of valid keywords for use in service function prototypes:

LIBRARYIndicates the name of the .SO module an entry point is defined in.
ENTRYPOINTSpecifies a name for the entry point. By default, the name of the entry point is the function name.
INITFUNCTIONSpecifies the name of the initialization routine defined in the module containing the entry point. Currently, the initialization function is called once.
INCLUDEIndicates the function prototype is in the specified include file, so the generated CPP must #include that file. If INCLUDE is not specified, the C++ prototype is generated from the ECL function definition.
CIndicates the generated C++ prototype is enclosed within an extern "C" rather than just extern.
PUREIndicates the function returns the same result every time you call it with the same parameters and has no side effects. This allows the optimizer to make more efficient calls to the function in some cases.
ONCEIndicates the function has no side effects and is evaluated at query execution time, even if the parameters are constant. This allows the optimizer to make more efficient calls to the function in some cases.
FOLDSpecifies that the function is evaluated at compile time if all parameters are constants. Specifying FOLD to the SERVICE applys it to all function definitions in the service - in such cases NOFOLD may be useful to override this default for individual functions that are not suitable for constant folding.
NOFOLDSpecifies that the service is not suitable for constant folding.
ACTIONIndicates the function has side effects and requires the optimizer to not remove calls to the function.
CONTEXTInternal use, only. Indicates an extra internal context parameter (ICodeContext *) is passed to the function. This must be the first function parameter.
GLOBALCONTEXTInternal use, only. Same as CONTEXT, but there are restrictions on where the function can be used (for example, not in a TRANSFORM).
CTXMETHODInternal use, only. Indicates the function is actually a method of the internal code context.

Data Types

Please see ECL to C++ Mapping documentation for data type mapping.

Passing Set Parameters to a Service

Three types of set parameters are supported: INTEGER, REAL, and STRINGn.

INTEGER

If you want to sum up all the elements in a set of integers with an external function, to declare the function in the SERVICE structure:

  SetFuncLib := SERVICE
    INTEGER SumInt(SET OF INTEGER ss) :
       holertl,library='dab',entrypoint='rtlSumInt';
  END;
  x:= 3+4.5;
  SetFuncLib.SumInt([x, 11.79]); //passed two REAL numbers - it works

To define the external function, in the header (.h) file:

__int64 rtlSumInt(unsigned len, __int64 * a);

In the source code (.cpp) file:

  __int64 rtlSumInt(unsigned len, __int64 * a) {
       __int64 sum = 0;
       for(unsigned i = 0; i < len; i++) {
            sum += a[i];
       }
       return sum;
    }

The first parameter contains the length of the set, and the second parameter is an int array that holds the elements of the set. Note: In declaring the function in ECL, you can also have sets of INTEGER4, INTEGER2 and INTEGER1, but you need to change the type of the C function parameter, too. The relationship is:

  INTEGER8 -- __int64
  INTEGER4 -- int
  INTEGER2 -- short
  INTEGER1 -- char

REAL

If you want to sum up all the elements in a set of real numbers:

To declare the function in the SERVICE structure:

  SetFuncLib := SERVICE
       REAL8 SumReal(SET OF REAL8 ss) :
            holertl,library='dab',entrypoint='rtlSumReal';
  END;
  
  INTEGER r1 := 10;
  r2 := 20.345;
  SetFuncLib.SumReal([r1, r2]);
  // intentionally passed an integer to the real set, it works too.

To define the external function, in the header (.h) file:

double rtlSumReal(unsigned len, double * a);

In the source code (.cpp) file:

  double rtlSumReal(unsigned len, double * a) {
    double sum = 0;
    for(unsigned i = 0; i < len; i++) {
       sum += a[i];
    }
    return sum;
  }

The first parameter contains the length of the set, and the second parameter is an array that holds the elements of the set.

Note: You can also declare the function in ECL as set of REAL4, but you need to change the parameter of the C function to float.

STRINGn

If you want to calculate the sum of the lengths of all the strings in a set, with the trailing blanks trimmed off:

To declare the function in the SERVICE structure:

  SetFuncLib := SERVICE
    INTEGER SumCharLen(SET OF STRING20 ss) :
       holertl,library='dab',entrypoint='rtlSumCharLen';
  END;
  str1 := '1234567890'+'xxxx ';
  str2 := 'abc';
  SetFuncLib.SumCharLen([str1, str2]);

To define the external function, in the header (.h) file:

__int64 rtlSumCharLen(unsigned len, char a[ ][20]);

In the source code (.cpp) file:

__int64 rtlSumCharLen(unsigned len, char a[][20]) {
    __int64 sumtrimedlen = 0;
       for(unsigned i = 0; i < len; i++) {
          for(int j = 20-1; j >= 0; j--) {
            if(a[i][j] != ' ') {
              break;
            }
            a[i][j] = 0;
       }
       sumtrimedlen += j + 1;
    }
    return sumtrimedlen;
  } 

Note: In declaring the C function, we have two parameters for the set. The first parameter is the length of the set, the second parameter is char[][n] where n is the SAME as that in stringn. Eg., if the service is declared as "integer SumCharLen(set of string20)", then in the C function the parameter type must be char a[][20].

Plugin Requirements

Plugins require an exported function with the following signature under Windows:

Extern "C" _declspec(dllexport) bool getECLPluginDefinition(ECLPluginDefinitionBlock *pb)

The function must fill the passed structure with correct information for the features of the plugin. The structure is defined as follows:

Warning: This function may be called without the plugin being loaded fully. It should not make any library calls or assume that dependent modules have been loaded or that it has been initialised. Specifically: "The system does not call DllMain for process and thread initialization and termination. Also, the system does not load additional executable modules that are referenced by the specified module."

Struct ECLPluginDefinitionBlock
  {
    Size_t size;
       //size of passed structure - filled in by the calling function
    Unsigned magicVersion ;
       // Filled in by .SO - must be PLUGIN_VERSION (1) 
    Const char *moduleName;
       // Name of the module 
    Const char *ECL;
       // ECL Service definition for non-HOLE applications
    Unsigned flags;
       // Type of plugin - for user plugin use 1
    Const char *version ;
       // Text describing version of plugin - used in debugging
    Const char *description;
       // Text describing plugin
  } 

To initialize information in a plugin, use a global variable or class and it will be appropriately constructed/destructed when the plugin is loaded and unloaded.

Deployment

External .SOs must be deployed to the /opt/HPCCSystems/plugins directory on each node of the target environment. If external data files are required, they should be either manually deployed to each node, or referenced from a network node (the latter requires hard-coding the address in the code for the .SO). Note that manually deployed files are not backed up with the standard SDS backup utilities.

Constraints

The full set of data types is supported on the Data Refinery and Data Delivery Engines (Thor/Roxie/Doxie).

An Example Service

The following code example depicts an ECL system service (.SO) called examplelib that contains one entry point (stringfind). This is a slightly modified version of the Find function found in the Str standard library. This version is designed to work in the Data Refinery supercomputer.

ECL definitions

  EXPORT ExampleLib := SERVICE
    UNSIGNED4 StringFind(CONST STRING src,
          CONST STRING tofind,
          UNSIGNED4 instance )
       : c, pure,entrypoint='elStringFind';
  END; 

.SO code module:

//******************************************************
  // hqlplugins.hpp : Defines standard values included
              in
  // the plugin header file.
  //******************************************************
  #ifndef __HQLPLUGIN_INCL
  #define __HQLPLUGIN_INCL
  
  #define PLUGIN_VERSION 1
  
  #define PLUGIN_IMPLICIT_MODULE 1
  #define PLUGIN_MODEL_MODULE 2
  #define PLUGIN_.SO_MODULE 4
  
  struct ECLPluginDefinitionBlock
  {
    size_t size;
    unsigned magicVersion;
    const char *moduleName;
    const char *ECL;
    const char *Hole;
    unsigned flags;
    const char *version;
    const char *description;
  };
  
  typedef bool (*EclPluginDefinition) (ECLPluginDefinitionBlock *);
  
  #endif //__HQLPLUGIN_INCL
//******************************************************
  // examplelib.hpp : Defines standard values included in
  // the plugin code file.
  //******************************************************
  #ifndef EXAMPLELIB_INCL
  #define EXAMPLELIB_INCL
  
  #ifdef _WIN32
    #define EXAMPLELIB_CALL __cdecl
    #ifdef EXAMPLELIB_EXPORTS
       #define EXAMPLELIB_API __declspec(dllexport)
    #else
       #define EXAMPLELIB_API __declspec(dllimport)
    #endif
  #else
    #define EXAMPLELIB_CALL
    #define EXAMPLELIB_API
  #endif
  
  #include "hqlplugins.hpp"
  
  extern "C" {
  EXAMPLELIB_API bool getECLPluginDefinition(ECLPluginDefinitionBlock *pb);
  EXAMPLELIB_API void setPluginContext(IPluginContext * _ctx);
  EXAMPLELIB_API unsigned EXAMPLELIB_CALL elStringFind(unsigned srcLen,
       const char * src, unsigned hitLen, const char * hit,
       unsigned instance);
  }
  
  #endif //EXAMPLELIB_INCL
  

//******************************************************
// examplelib.cpp : Defines the plugin code.
//******************************************************
#include <time.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include "examplelib.hpp"

#define EXAMPLELIB_VERSION "EXAMPLELIB 1.0.00"

static const char * HoleDefinition = NULL;

static const char * EclDefinition =
"export ExampleLib := SERVICE\n"
"  string EchoString(const string src) : c, pure,fold,entrypoint='elEchoString'; \n"
"END;";

EXAMPLELIB_API bool getECLPluginDefinition(ECLPluginDefinitionBlock *pb) 
{
    //  Warning:    This function may be called without the plugin being loaded fully.  
    //              It should not make any library calls or assume that dependent modules
    //              have been loaded or that it has been initialised.
    //
    //              Specifically:  "The system does not call DllMain for process and thread 
    //              initialization and termination.  Also, the system does not load 
    //              additional executable modules that are referenced by the specified module."

    if (pb->size != sizeof(ECLPluginDefinitionBlock))
        return false;

    pb->magicVersion = PLUGIN_VERSION;
    pb->version = EXAMPLELIB_VERSION " $Revision: 62376 $";
    pb->moduleName = "lib_examplelib";
    pb->ECL = EclDefinition;
    pb->Hole = HoleDefinition;
    pb->flags = PLUGIN_IMPLICIT_MODULE;
    pb->description = "ExampleLib example services library";
    return true;
}

namespace nsExamplelib {
    IPluginContext * parentCtx = NULL;
}
using namespace nsExamplelib;

EXAMPLELIB_API void setPluginContext(IPluginContext * _ctx) { parentCtx = _ctx; }

//-------------------------------------------------------------------------------------------------------------------------------------------

EXAMPLELIB_API unsigned EXAMPLELIB_CALL elStringFind(unsigned srcLen,
 const char * src, unsigned hitLen, const char * hit,
 unsigned instance)
{
    tgt = (char *)CTXMALLOC(parentCtx, srcLen);
    memcpy(tgt,src,srcLen);
    tgtLen = srcLen;
}