ECL IDE not compiling / submitting work units

[amillar]

Hi There,

today we upgraded our HPCC cluster from 7.6.22-1 to 7.8.26-1 and everything seemed to go without issues.

However, we are having problems submitting Work Units from the ECL IDE.

No matter what we submit we always get the same error :

WARNING: SOAP 1.1 fault: SOAP-ENV:Client[no subcode]
"An HTTP processing error occurred"
Detail: [no detail]
Repository.cpp(350)
INFO: eclcc.exe relative path: C:\Program Files (x86)\HPCCSystems\7.8.26\clienttools\bin\eclcc.exe EclCC.cpp(791)
INFO: eclcc.exe relative path: C:\Program Files (x86)\HPCCSystems\7.8.26\clienttools\bin\eclcc.exe EclCC.cpp(791)
INFO: eclcc.exe relative path: C:\Program Files (x86)\HPCCSystems\7.8.26\clienttools\bin\eclcc.exe EclCC.cpp(791)
INFO: eclcc.exe relative path: C:\Program Files (x86)\HPCCSystems\7.8.26\clienttools\bin\eclcc.exe


I have installed ECL IDE V 7.8.26-1 to match the cluster version but that has not solved the problem.

We are able to "clone" exiting work units which tells me the cluster is working.

I have attached a screen shot of the error, and you can see it doesn't seem to even compile.

Any help would be greatly appreciated.

Thanks in advance

Antony

[hwatanuki]

Hello amillar,

I have just installed a virtual cluster by using v7.8.26-1 of the platform and I do confirm that I am receiving the same informational warnings as you reported above in the ECL IDE logs panel. However, I don´t think they are causing the issues you are reporting with WU submission, as I am able to submit WU´s sucessfully despite the messages. The SOAP warning may actually be caused by a missing Attribute Server in your environment and, in this case, the warning can be eliminated by cleaning up/correcting the "Attribute Server" field in the ECL IDE preferences tab.

By the description of your scenario, I would suggest that you take a look at the "Operations>System Servers" tab in your ECL Watch and try to spot something wrong with the ECLCC Server status or in its log files. I was able to partially replicate the issue you describe by manually killing my eclcc daemon and trying to submit a job. Also, note that "cloning" a job from the ECL Watch doesn´t reproduce all the job submission workflow in its entirety, so it cannot be considered, in its essence, an "apples to apples" comparison to confirm that all components on the cluster side are working properly. Submitting a test job from the ECL playground panel in ECL Watch could give you a better data point in this specific case.

HTH,
HugoW

[gsmith]

As @hwatanuki points out, the "Warning" and "Info" logs you are seeing are probably unrelated to the underlying issue.

In ECL Watch can you open the ECL Playground and submit a Job to each of the target clusters (hthor, thor etc.) and see if that works?

[amillar]

Hi Hugo and G Smith,

Thanks for the quick reply,

I will definitely check the preferences in the ECL IDE to see if I can solve those warnings.

I have had a look at the operations - system services and I cannot see any issues with the ECL CC server.

Looking at the logs, the only warning / errors I receive are :

00000A5F PRG 2020-07-08 09:11:19.445 13998 49953 "Compile request processing for workunit W20200708-091117"
00000A60 USR 2020-07-08 09:11:19.452 13998 49953 "/opt/HPCCSystems/bin/eclcc: Creating PIPE program process : '/opt/HPCCSystems/bin/eclcc -shared --daemon myeclccserver - --timings --xml --nostdinc --metacache= --logfile=W20200708-091117.eclcc.log -dfs=192.168.24.120:7070 -cluster=hthor -wuid=W20200708-091117 -token=HPCC[u=,w=W20200708-091117]; -oW20200708-091117 -platform=hthor [email protected] -fcreated_by=ws_workunits' - hasinput=1, hasoutput=0 stderrbufsize=0"
00000A61 PRG 2020-07-08 09:11:19.478 13998 41355 "ERROR: Unrecognised error: Error: unrecognised option --daemon"
00000A62 PRG 2020-07-08 09:11:19.478 13998 49953 "ERROR: Unrecognised error: Pipe: write failed (size 368), Broken pipe"
00000A63 USR 2020-07-08 09:11:19.478 13998 49953 "/opt/HPCCSystems/bin/eclcc: Pipe: process 41354 complete 1"
00000A64 PRG 2020-07-08 09:11:19.538 13998 14211 "SYS: LPT=2 APT=96 PU= 0% MU= 0% MAL=108883968 MMP=108150784 SBK=733184 TOT=106512K RAM=2642524K SWP=0K"
00000A65 PRG 2020-07-08 09:11:19.538 13998 14211 "DSK: [sda] r/s=0.0 kr/s=0.0 w/s=0.7 kw/s=3.9 bsy=0 NIC: [bond0] rxp/s=38.3 rxk/s=5.4 txp/s=36.6 txk/s=12.1 rxerrs=0 rxdrps=53 txerrs=0 txdrps=0 CPU: usr=0 sys=0 iow=0 idle=99"
00000A66 PRG 2020-07-08 09:11:31.670 13998 49953 "Compile request processing for workunit W20200708-091130"
00000A67 USR 2020-07-08 09:11:31.679 13998 49953 "/opt/HPCCSystems/bin/eclcc: Creating PIPE program process : '/opt/HPCCSystems/bin/eclcc -shared --daemon myeclccserver - --timings --xml --nostdinc --metacache= --logfile=W20200708-091130.eclcc.log -dfs=192.168.24.120:7070 -cluster=Thor-Large -wuid=W20200708-091130 -token=HPCC[u=,w=W20200708-091130]; -oW20200708-091130 -platform=thorlcr [email protected] -fcreated_by=ws_workunits' - hasinput=1, hasoutput=0 stderrbufsize=0"
00000A68 PRG 2020-07-08 09:11:31.702 13998 41370 "ERROR: Unrecognised error: Error: unrecognised option --daemon"
00000A69 PRG 2020-07-08 09:11:31.702 13998 49953 "ERROR: Unrecognised error: Pipe: write failed (size 368), Broken pipe"
00000A6A USR 2020-07-08 09:11:31.702 13998 49953 "/opt/HPCCSystems/bin/eclcc: Pipe: process 41369 complete 1"
00000A6B PRG 2020-07-08 09:11:39.493 13998 49953 "Compile request processing for workunit W20200708-091139"
00000A6C USR 2020-07-08 09:11:39.500 13998 49953 "/opt/HPCCSystems/bin/eclcc: Creating PIPE program process : '/opt/HPCCSystems/bin/eclcc -shared --daemon myeclccserver - --timings --xml --nostdinc --metacache= --logfile=W20200708-091139.eclcc.log -dfs=192.168.24.120:7070 -cluster=Thor-Small -wuid=W20200708-091139 -token=HPCC[u=,w=W20200708-091139]; -oW20200708-091139 -platform=thorlcr [email protected] -fcreated_by=ws_workunits' - hasinput=1, hasoutput=0 stderrbufsize=0"
00000A6D PRG 2020-07-08 09:11:39.525 13998 41383 "ERROR: Unrecognised error: Error: unrecognised option --daemon"
00000A6E PRG 2020-07-08 09:11:39.525 13998 49953 "ERROR: Unrecognised error: Pipe: write failed (size 368), Broken pipe"
00000A6F USR 2020-07-08 09:11:39.525 13998 49953 "/opt/HPCCSystems/bin/eclcc: Pipe: process 41382 complete 1"


I have tried submitting a job from the ECL playground against all targets and unfortunately that has not been successful.

If it helps we are running the platform on Ubuntu 18.04.

Hopefully the information in the log above will help narrow this down

Thanks in advance
Antony

[gsmith]

I have tried submitting a job from the ECL playground against all targets and unfortunately that has not been successful.

This rules out any issue with the ECLIDE and the Info / Warning messages can be safely ignored.

[amillar]

Thanks for the quick reply,

agreed this seems to be a problem with the cluster, do those log files help?

Do you need anymore information from me to help narrow down the problem?

Thanks

Antony

[ghalliday]

This looks like it is probably a platform regression. I think I know the fix, but I am confused why it wasn't caught much earlier in testing.

[ghalliday]

See https://track.hpccsystems.com/browse/HPCC-24389

The problem only appears if you use systemctl to start the system. You can use the old scripts in /etc/init.d/hpcc-init to start as a work around. Fix should be in the next point release.

[amillar]

Hi ghalliday.

thanks for your help, you are absolutely correct switching from systemctl to init.d has solved our problem.

Best Regards

Antony