Sun Nov 17, 2019 6:20 pm
Login Register Lost Password? Contact Us


Call REST POST from ECL

Comments and questions related to the Enterprise Control Language

Mon Aug 19, 2019 5:04 pm Change Time Zone

I see from the help that HTTPCALL only supports GET and that there is an outstanding ticket to add the ability to call POST https://track.hpccsystems.com/browse/HPCC-9805.

My question is what is the workaround for this while we are waiting for this to be supported?

Can a C++ ECL function be written to do this? Has anyone got an example of the C++ code I need to implement in an ECL function to do this?

Can curl be called from the built in PIPE function or CmdProcess function?

Thanks

Ben
brw
 
Posts: 4
Joined: Mon Aug 19, 2019 4:57 pm

Mon Aug 19, 2019 5:50 pm Change Time Zone

Hi Ben,

If you're comfortable with C++ access to curl (technically, libcurl) then the following my give you a leg up on writing your own embedded function:

Code: Select all
IMPORT Std;

#WORKUNIT('name', 'libcurl testing');

STRING CallCurl(CONST VARSTRING url) := EMBED(C++)
    #option library curl

    #include <curl/curl.h>

    struct MemoryStruct
    {
        char*   memory;
        size_t  size;
        size_t  capacity;
    };

    static void InitMemoryStructCapacity(MemoryStruct& mem, size_t initCapacity)
    {
        mem.memory = static_cast<char*>(rtlMalloc(initCapacity));
        mem.size = 0;
        mem.capacity = initCapacity;
    }

    static size_t CaptureIncomingReply(void* contents, size_t size, size_t nmemb, void* userp)
    {
        size_t          incomingDataSize = size * nmemb;
        MemoryStruct*   mem = static_cast<struct MemoryStruct*>(userp);

        if (mem->size + incomingDataSize > mem->capacity)
        {
            size_t  newCapacity = mem->capacity * 2;

            // Keep doubling capacity until it is greater than what we need
            while (mem->size + incomingDataSize > newCapacity)
            {
                newCapacity *= 2;
            }

            mem->memory = static_cast<char*>(rtlRealloc(mem->memory, newCapacity));
            mem->capacity = newCapacity;
        }

        memcpy(&(mem->memory[mem->size]), contents, incomingDataSize);
        mem->size += incomingDataSize;
       
        return incomingDataSize;
    }

    #body

    __lenResult = 0;
    __result = NULL;

    CURL*   curlHandle = curl_easy_init();

    if(curlHandle)
    {
        CURLcode            curlResponseCode;
        struct curl_slist*  headers = NULL;
        MemoryStruct        captureBuffer;

        // Initialize our capture buffer to a reasonable size to avoid
        // memory reallocation
        InitMemoryStructCapacity(captureBuffer, 8196);

        // headers = curl_slist_append(headers, "Content-Type: application/json");
        // curl_easy_setopt(curlHandle, CURLOPT_HTTPHEADER, headers);

        curl_easy_setopt(curlHandle, CURLOPT_URL, url);
        curl_easy_setopt(curlHandle, CURLOPT_FOLLOWLOCATION, 1);
        curl_easy_setopt(curlHandle, CURLOPT_NOPROGRESS, 1);
        curl_easy_setopt(curlHandle, CURLOPT_WRITEFUNCTION, CaptureIncomingReply);
        curl_easy_setopt(curlHandle, CURLOPT_WRITEDATA, static_cast<void*>(&captureBuffer));

        curlResponseCode = curl_easy_perform(curlHandle);

        if (curlResponseCode == CURLE_OK)
        {
            long    httpResponseCode;

            curl_easy_getinfo(curlHandle, CURLINFO_RESPONSE_CODE, &httpResponseCode);

            if (httpResponseCode == 200 && captureBuffer.size > 0)
            {
                __lenResult = captureBuffer.size;
                __result = captureBuffer.memory;
            }
        }

        curl_easy_cleanup(curlHandle);
        curl_slist_free_all(headers);
    }
ENDEMBED;

//-------

CallCurl('http://example.com');

This was extracted from a running project that uses libcurl for REST access. As written, it uses GET but obviously you can make curl execute a POST instead. I've left a couple of commented-out around (populating headers) that illustrate where that goes as well.

Oh, it does assume that you have a developer's version of libcurl installed (meaning, both the library and header files).

Hope this helps.

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 566
Joined: Tue Oct 18, 2011 4:45 pm

Tue Aug 20, 2019 10:54 am Change Time Zone

Thanks Dan,

I managed to get a test working in my Local VM HPCC using the CmdProcess function (see ECL below).

What are the Pros and Cons of this approach compared to the embedded C++ solution?
Code: Select all
cmd := 'curl -X POST http://10.243.5.1:8080/journey_lookup -H "Content-Type: application/json"';

Body := '{"Journeys":[{"JourneyId":1206582,"Id":[1826],"XLongitude":[-84.22373],"YLatitude":[34.15089],"HorizontalSpeed":"OA==","ReverseGeoCodeRow":null,"PulseDateTimeUTCAsUnixEpoch":null,"RoadSpeed":null,"RoadSegId":null,"RoadTypeId":null,"Validated":null,"CountryId":null,"RoadCategory":[0],"BoundaryId":null,"NearestRoadDistance":null,"Proximity":null,"NearestRoadSegId":null}],"CalculateRailwayProximity":false,"RoadSpeedRule":"SmartPhone"}';

STD.System.Util.CmdProcess(cmd + ' -d \'' + Body + '\'', '');
brw
 
Posts: 4
Joined: Mon Aug 19, 2019 4:57 pm

Tue Aug 20, 2019 11:38 am Change Time Zone

The embedded version is far more performant. ECL is transpiled to C++, so there are no external environments or context switching involved with using an embedded C++ function. The function is simply included in the generated C++ source code.

Std.System.Util.CmdProcess() and PIPE() invoke external applications by forking the current process and executing the binary in place of the fork. Data I/O has to be serialized, parsed, etc. There is more work and more system resources involved for every invocation.

If you're processing a lot of records then it will definitely pay off in the long run to use the libcurl version.

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 566
Joined: Tue Oct 18, 2011 4:45 pm

Wed Aug 21, 2019 4:55 pm Change Time Zone

Thanks again Dan for the C++, but unfortunately it doesn't work on my environment presumably because I don't have libcurl dependencies installed and modifying our environment to add them is probably not an option.

Is there a way to do this without the need to install any additional libraries - ie with just the libraries that come with HPCC as standard?

Which Libraries does the built in HPCC HTTPCall function use under the covers to call a REST GET, can I use the same libraries in a C++ HPCC function?
brw
 
Posts: 4
Joined: Mon Aug 19, 2019 4:57 pm

Wed Aug 21, 2019 7:13 pm Change Time Zone

Unfortunately, I believe the platform uses internal code (read: no external libraries) for TCP/IP communication. Leveraging that is possible, but it would actually require an even bigger change to your dev environment because you would have to install the platform source code instead of just the libcurl headers. Given the choice, libcurl would be the smaller installation by a long shot.

(For clarity: You would need to install the dev libcurl stuff only on your HPCC Systems node that is running eclccserver. You can get away with installing the non-dev/regular libcurl on all other HPCC Systems nodes. libcurl's headers are required only for compilation of your workunit.)

If you cannot change the environment then you may need to use Std.System.Util.CmdProcess() or PIPE() and invoke the curl binary to do what you need. Hopefully your distribution has the curl binary already installed, and in a standard location.
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 566
Joined: Tue Oct 18, 2011 4:45 pm

Wed Aug 21, 2019 8:52 pm Change Time Zone

You might consider using embedded JavaScript (v8). I haven't tried it myself, but it may have some pretty convenient HTTP related functions built in.

You'll have to figure out the HTTP call stuff elsewhere, but you can find some simple embedded Javascript test cases here:

https://github.com/hpcc-systems/HPCC-Pl ... mbedjs.ecl

Python and JAVA are other options.

As mentioned, we do use our own internal C++ HTTP classes so you would need the headers and link libraries in place to use those. There are other c++ HTTP libraries you could use but would require installing dependencies.
anthony.fishbeck
 
Posts: 57
Joined: Wed Jan 30, 2013 10:18 pm

Thu Aug 22, 2019 3:12 pm Change Time Zone

Thanks everyone.

It turns out to be pretty simple implementation in Python, I hope this helps someone else who is also struggling with the lack of built in Support for REST calls :D

In my case I am POSTing a semicolon delimited String containing muliple records and the Service returns a semicolon delimited String with a different layout to the Request.

NOTE: Response_Layout has all fields as STRING, whereas JourneysIn_Layout has the correct types, so there is some subsequent code to project the result to a Typed Dataset, which I haven't shown here.

Code: Select all
DATASET(Response_Layout) CallREST(STRING IP, UNSIGNED2 Port, DATASET(JourneysIn_Layout) JourneysIn) := EMBED(Python)

   import httplib
   import base64
   import ssl
   import csv
   import StringIO

   conn = httplib.HTTPConnection(IP, Port)

   headers = {}
   headers['Content-Type'] = "text/csv"
   #headers['Authorization'] = \
   #    "Basic %s" % base64.standard_b64encode("admin:ibm")

   StrWriter = StringIO.StringIO()

   BodyWriter = csv.writer(StrWriter, delimiter=';')

   for Journey in JourneysIn:
      BodyWriter.writerow(Journey)

   Body = StrWriter.getvalue()

   req = conn.request('POST', '/api/odin/enrich_journeys/hpcc', headers=headers, body=Body)
   res = conn.getresponse()

   csvLines = csv.reader(res.read().splitlines(), delimiter=';')
   for csvLine in csvLines:
         yield tuple(csvLine)

ENDEMBED;
brw
 
Posts: 4
Joined: Mon Aug 19, 2019 4:57 pm

Thu Aug 22, 2019 3:46 pm Change Time Zone

Excellent. Thank you for posting your example. It's very helpful.
anthony.fishbeck
 
Posts: 57
Joined: Wed Jan 30, 2013 10:18 pm


Return to ECL

Who is online

Users browsing this forum: No registered users and 0 guests

cron