Fri Aug 17, 2018 3:19 am
Login Register Lost Password? Contact Us


High throughput Roxie clusters

Topics related to recommendations or questions on the design for HPCC Systems clusters

Sat May 18, 2013 1:04 am Change Time Zone

I am setting up a high throughput Roxie cluster but it seems like the daserver process on the master node is the bottleneck.

I have turned up the logging on my Roxie servers and they now record the resource stats for each query. I can see from the Roxie logs that the query needed only 10 ms to execute. However, the end to end turnaround time for the query (executed on localhost) was 60 ms. On some more investigation, I found that the extra time was for the post processing by daserver and esp processes. I was successful in setting up multiple instances for esp and load balancing externally between those ESP instances, but I am not sure if that helped. Sometimes my ESP servers will just stop communicating new requests to Roxie until Dali is restarted. In the ESP logs, I see the following lines:
Code: Select all
00000302 2013-05-17 20:52:32 20428 23613 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."

followed by
Code: Select all
00000680 2013-05-17 20:57:31 20428 23614 "Sending SOAP Fault(1315): <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/04/secext"><soap:Body><soap:Fault><faultcode>1</faultcode><faultstring>[  1: SDS: Lock timeout
SDS Reply Error  : SDS: Lock timeout
Failed to establish lock to QuerySets/
Existing lock status: Locks on path: /QuerySets/
Endpoint            |SessionId       |ConnectionId    |mode    |time(duration)]

10.38.251.1:7234    |630000042a      |6300009cf8      |242     |2013-05-18T00:52:33(298986 ms)
] </faultstring><faultactor>Esp</faultactor><detail><Exceptions xmlns="urn:hpccsystems:ecl:soap" xsi:schemaLocation="urn:hpccsystems:ecl:soap http://10.38.251.1:8002/WsEcl/soap?xsd"><Source>Esp</Source><Exception><Code>1</Code><Audience>user</Audience><Message>SDS: Lock timeout
SDS Reply Error  : SDS: Lock timeout
Failed to establish lock to QuerySets/
Existing lock status: Locks on path: /QuerySets/
Endpoint            |SessionId       |ConnectionId    |mode    |time(duration)]

10.38.251.1:7234    |630000042a      |6300009cf8      |242     |2013-05-18T00:52:33(298986 ms)
</Message></Exception></Exceptions></detail></soap:Fault></soap:Body></soap:Envelope>"


What are the best practices around setting up a high throughput Roxie cluster? It seems like the daserver process is a botleneck at the moment.

My HPCC version is 3.10.8-3.
sbagaria
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 92
Joined: Thu Jul 05, 2012 9:23 pm

Mon May 20, 2013 1:12 pm Change Time Zone

Resolved with a hint from Richard Chapman. wsECL on port 8002 is to be used strictly for testing. For high performance systems, we should use Roxie directly on port 9876.

For example, if previously the SOAP query URL looked like this
http://localhost:8002/wsECL/soap/query/roxie/queryname
it should now look like this
http://localhost:9876/

The actual SOAP query and response will remain the same.
sbagaria
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 92
Joined: Thu Jul 05, 2012 9:23 pm

Mon May 20, 2013 1:25 pm Change Time Zone

This also means that you don't need multiple ESP instances. You can query any of the Roxie servers on your cluster directly. This can be put behind an external load balancer if you wish.
sbagaria
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 92
Joined: Thu Jul 05, 2012 9:23 pm

Mon May 20, 2013 1:30 pm Change Time Zone

sbagaria
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 92
Joined: Thu Jul 05, 2012 9:23 pm

Wed May 22, 2013 2:18 pm Change Time Zone

Thank you for sharing the post and how it was resolved.
admin
Site Admin
Site Admin
 
Posts: 203
Joined: Thu Jan 27, 2011 10:58 am

Tue Sep 17, 2013 7:41 pm Change Time Zone

sbagaria wrote:Resolved with a hint from Richard Chapman. wsECL on port 8002 is to be used strictly for testing. For high performance systems, we should use Roxie directly on port 9876.

For example, if previously the SOAP query URL looked like this
http://localhost:8002/wsECL/soap/query/roxie/queryname
it should now look like this
http://localhost:9876/

The actual SOAP query and response will remain the same.


What does a URL look like for querying a Roxie server directly with JSON, with v4.0 of HPCC? The RDDERef documentation leaves out that important factoid.

Thanks,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 552
Joined: Tue Oct 18, 2011 4:45 pm

Tue Sep 17, 2013 7:44 pm Change Time Zone

I have not tried with v4.0. But for previous versions, just send your SOAP query on port 9876 without any path.

AFAIK, there is no JSON support when querying Roxie directly. ESP provides the additional JSON layer to Roxie. I don't think this has changed in v4.0.
sbagaria
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 92
Joined: Thu Jul 05, 2012 9:23 pm

Tue Sep 17, 2013 7:49 pm Change Time Zone

I think that v4.0 does support it. The RDDERef-4.0.0-9 manual talks about it (page 37) and the release notes cite it ("HPCC-8953 Add roxie support for JSON/HTTP queries").

I've used the direct SOAP interface without a problem. I'm hoping to do the same thing with JSON.

Thanks,

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 552
Joined: Tue Oct 18, 2011 4:45 pm

Tue Sep 17, 2013 8:59 pm Change Time Zone

Sending JSON requests directly to roxie is supported in 4.0. The trick is not in the url used but rather in setting the HTTP content type to "application/json" and formatting the JSON request correctly.

Even though you're not planning on using WsEcl in production, it can still be a great
development and QA tool, showing you the format of the request you should send to roxie and allowing you to test your own requests, etc.

Navigate to the WsEcl form for your query. Fill in any values you would like populated and change the drop down box below the form to read "JSON Test" and click submit.

The Request box should be populated with a well formatted JSON request. Click submit to see what the roxie response will look like.

FYI In the latest release sending JSON and SOAP requests to roxie through WsEcl has also been optimized, minimizing the overhead. It no longer gets workunit resources from DALI for pre and post processing. So if you get benefit from using WsEcl you can reconsider using it for SOAP and JSON requests.
anthony.fishbeck
 
Posts: 47
Joined: Wed Jan 30, 2013 10:18 pm

Tue Sep 17, 2013 9:04 pm Change Time Zone

To summarize to make sure I understand: The URLs for both SOAP and JSON requests are identical. Roxie examines the HTTP content type to determine how to both parse the request and generate the response. It Just Works.

Is that right?
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 552
Joined: Tue Oct 18, 2011 4:45 pm

Next

Return to Clustering

Who is online

Users browsing this forum: No registered users and 1 guest

cron