Mon Oct 25, 2021 1:02 am
Login Register Lost Password? Contact Us


Workunit running for long time!!

Topics specific to the use of the ECL IDE

Thu Feb 14, 2013 8:36 am Change Time Zone

Hi,

Our general usage scenario is :

    The general requirement is that a sub-file will be added every n minutes to a superfile and I need an index to be built on this superfile. Multiple such super-files(hence, sub-files exist)
    A precreated superfile viz. ~prachi::superfile exists on Thor
    Via a web application ->ECL code on HPCC master, multiple users are adding sub-files to super-files other than ~prachi::superfile and also reading the super-files via the HPCC-JDBC driver

Now, I'm trying to execute the following ECL code via the ECL IDE, multiple times:

THOR_CODE:

Code: Select all
IMPORT Std;

SuperFile_name := '~prachi::superfile';
indexfile_name := '~prachi::indexfile';

VARSTRING fileName := 'clientwebcrawldata_2';
VARSTRING destinationlogicalname  := '~sprayed::' +fileName;
VARSTRING sourceIP := '10.101.2.171';
VARSTRING sourcepath := '/var/lib/HPCCSystems/mydropzone/superfile_files/' +fileName +'.csv';
VARSTRING srcCSVseparator := ';';
VARSTRING destinationgroup := 'mythor';
VARSTRING espserverIPport := 'http://10.101.2.170:8010/FileSpray';
VARSTRING subFileDestinationLogicalname := '~prachi::subfile::' +fileName;

/*Spray the csv file from the dropzone*/
SprayCSVFile :=STD.File.fSprayVariable(sourceIP,sourcepath,,srcCSVseparator,,,
destinationgroup,destinationlogicalname,,espserverIPport,
,TRUE,TRUE,FALSE);


Layout_ClientWebCrawlData := RECORD
VARSTRING Controller_ID;
VARSTRING User_ID;
VARSTRING URL_Link;
VARSTRING URL_Content;
REAL URL_Sentiment;
VARSTRING URL_Date;
INTEGER8 Unique_Search_ID;
INTEGER8 InsertID;
VARSTRING StatusID;
VARSTRING Search_Pattern;
VARSTRING Word_Ignored;
VARSTRING Search_Date;
VARSTRING Detected_Language;
END;

File_Layout_logical_file_Dataset := DATASET(destinationlogicalname,Layout_ClientWebCrawlData,CSV(SEPARATOR(';')));

subfileCreation := OUTPUT(File_Layout_logical_file_Dataset,,subFileDestinationLogicalname,THOR,OVERWRITE);

deleteSprayedLogicalFile := STD.File.DeleteLogicalFile(destinationlogicalname);

SuperFile_Dataset := DATASET(SuperFile_name,{Layout_ClientWebCrawlData,UNSIGNED8 fpos {virtual(fileposition)}},THOR);

IDX_SuperFile := INDEX(SuperFile_Dataset,{InsertID,fpos},indexfile_name);
build_superfile_index := BUILDINDEX(IDX_SuperFile,OVERWRITE);

SEQUENTIAL(
SprayCSVFile,
subfileCreation,
deleteSprayedLogicalFile,
Std.File.StartSuperFileTransaction(),
Std.File.AddSuperFile(SuperFile_name,subFileDestinationLogicalname),
Std.File.FinishSuperFileTransaction(),
build_superfile_index
);


These steps are performed correctly when submitted for first time i.e the 1st subfile is added and the index also is created correctly.

Respective to this code an ECL code to compile on ROXIE has been written which publishes successfully and is also giving results(via WsECL).

ROXIE_CODE :

Code: Select all
IMPORT STD;

Export QueryToRunOnRoxie():=Function
INTEGER insertid_value := 0 : STORED('insertid');

Layout_ClientWebCrawlData := RECORD
VARSTRING Controller_ID;
VARSTRING User_ID;
VARSTRING URL_Link;
VARSTRING URL_Content;
REAL URL_Sentiment;
VARSTRING URL_Date;
INTEGER8 Unique_Search_ID;
INTEGER8 InsertID;
VARSTRING StatusID;
VARSTRING Search_Pattern;
VARSTRING Word_Ignored;
VARSTRING Search_Date;
VARSTRING Detected_Language;
END;
   
STRING SuperFile_name := '~prachi::superfile';
STRING indexfile_name := '~prachi::indexfile';

// create dataset of superfile
SuperFile_Dataset := DATASET(SuperFile_name,{Layout_ClientWebCrawlData,UNSIGNED8 fpos {virtual(fileposition)}},THOR);

// create index on superfile
IDX_SuperFile := INDEX(SuperFile_Dataset,{InsertID,fpos},indexfile_name);

//fetch records
fetched_records := FETCH(SuperFile_Dataset, IDX_SuperFile(insertid=insertid_value), RIGHT.fpos);

//store output
getOutput := OUTPUT(fetched_records);

RETURN getOutput;
END;


But when THOR_CODE runs again, say 2nd time, the workunit goes in the 'running' state for an indefinite period. I suspect that though I'm working on a super-file not used by anyone, some deadlock/blocking is occurring at Thor in 4 node cluster. But when tried on separate single-node cluster, with only above two codes (no web app is running.This cluster is totally separate), same issue takes place.
Please evaluate my assumption.

Eclagent.log of single-node cluster is as follows:

Code: Select all
00000000 2013-02-14 20:40:02 30752 30752 Logging to /var/log/HPCCSystems/myeclagent/eclagent.2013_02_14.log
00000001 2013-02-14 20:40:02 30752 30752 ECLAGENT build community_3.10.0-1
00000002 2013-02-14 20:40:02 30752 30752 Waiting for workunit lock
00000003 2013-02-14 20:40:02 30752 30752 Obtained workunit lock
00000004 2013-02-14 20:40:02 30752 30752 Loading dll (libW20130214-203956.so) from location /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so
00000005 2013-02-14 20:40:02 30752 30752 Starting process
00000006 2013-02-14 20:40:02 30752 30752 RoxieMemMgr: Setting memory limit to 314572800 bytes (300 pages)
00000007 2013-02-14 20:40:02 30752 30752 RoxieMemMgr: 320 Pages successfully allocated for the pool - memsize=335544320 base=0x7f521bf00000 alignment=1048576 bitmapSize=10
00000008 2013-02-14 20:40:02 30752 30752 Waiting for run lock
00000009 2013-02-14 20:40:02 30752 30752 Obtained run lock
0000000A 2013-02-14 20:40:02 30752 30752 Spray:  ~sprayed::clientwebcrawldata_2
0000000B 2013-02-14 20:40:09 30752 30752 setResultString((null),0,'D20130214-204002')
0000000C 2013-02-14 20:40:09 30752 30752 Enqueuing on thor.thor to run wuid=W20130214-203956, graph=graph1, timelimit=600 seconds, priority=0
0000000D 2013-02-14 20:40:09 30752 30752 Thor on 172.25.37.135:20000 running W20130214-203956
0000000E 2013-02-14 20:40:10 30752 30752 Deleting NS logical file sprayed::clientwebcrawldata_2 for user 10602210
0000000F 2013-02-14 20:40:10 30752 30752 DeleteLogicalFile ('sprayed::clientwebcrawldata_2') done
00000010 2013-02-14 20:40:10 30752 30752 ,FileAccess,FileServices,DeleteLogicalFile,W20130214-203956,10602210,sprayed::clientwebcrawldata_2
00000011 2013-02-14 20:40:10 30752 30752 StartSuperFileTransaction
00000012 2013-02-14 20:40:10 30752 30752 AddSuperFile ('prachi::superfile', 'prachi::subfile::clientwebcrawldata_2') trans
00000013 2013-02-14 20:40:10 30752 30752 ,FileAccess,FileServices,AddSuperFile,W20130214-203956,10602210,prachi::superfile,prachi::subfile::clientwebcrawldata_2
00000014 2013-02-14 20:40:15 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000015 2013-02-14 20:40:20 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000016 2013-02-14 20:40:20 30752 30752 Backtrace:
00000017 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000018 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000019 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000001A 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000001B 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000001C 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000001D 2013-02-14 20:40:20 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
0000001E 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
0000001F 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000020 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000021 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000022 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000023 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000024 2013-02-14 20:40:20 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000025 2013-02-14 20:40:20 30752 30752   eclagent(main+0x51) [0x400f41]
00000026 2013-02-14 20:40:20 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000027 2013-02-14 20:40:20 30752 30752   eclagent() [0x401019]
00000028 2013-02-14 20:40:20 30752 30752 CDFAction lock timed out on prachi::superfile
00000029 2013-02-14 20:40:20 30752 30752 CDistributedFileTransaction: Transaction pausing
0000002A 2013-02-14 20:41:53 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
0000002B 2013-02-14 20:41:58 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000002C 2013-02-14 20:41:58 30752 30752 Backtrace:
0000002D 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000002E 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000002F 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000030 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000031 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000032 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000033 2013-02-14 20:41:58 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000034 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000035 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000036 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000037 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000038 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000039 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
0000003A 2013-02-14 20:41:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
0000003B 2013-02-14 20:41:58 30752 30752   eclagent(main+0x51) [0x400f41]
0000003C 2013-02-14 20:41:58 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
0000003D 2013-02-14 20:41:58 30752 30752   eclagent() [0x401019]
0000003E 2013-02-14 20:41:58 30752 30752 CDFAction lock timed out on prachi::superfile
0000003F 2013-02-14 20:41:58 30752 30752 CDistributedFileTransaction: Transaction pausing
00000040 2013-02-14 20:43:05 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000041 2013-02-14 20:43:10 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000042 2013-02-14 20:43:10 30752 30752 Backtrace:
00000043 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000044 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000045 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000046 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000047 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000048 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000049 2013-02-14 20:43:10 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
0000004A 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
0000004B 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
0000004C 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
0000004D 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
0000004E 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
0000004F 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000050 2013-02-14 20:43:10 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000051 2013-02-14 20:43:10 30752 30752   eclagent(main+0x51) [0x400f41]
00000052 2013-02-14 20:43:10 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000053 2013-02-14 20:43:10 30752 30752   eclagent() [0x401019]
00000054 2013-02-14 20:43:10 30752 30752 CDFAction lock timed out on prachi::superfile
00000055 2013-02-14 20:43:10 30752 30752 CDistributedFileTransaction: Transaction pausing
00000056 2013-02-14 20:44:16 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000057 2013-02-14 20:44:21 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000058 2013-02-14 20:44:21 30752 30752 Backtrace:
00000059 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000005A 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000005B 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000005C 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000005D 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000005E 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000005F 2013-02-14 20:44:21 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000060 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000061 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000062 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000063 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000064 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000065 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000066 2013-02-14 20:44:21 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000067 2013-02-14 20:44:21 30752 30752   eclagent(main+0x51) [0x400f41]
00000068 2013-02-14 20:44:21 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000069 2013-02-14 20:44:21 30752 30752   eclagent() [0x401019]
0000006A 2013-02-14 20:44:21 30752 30752 CDFAction lock timed out on prachi::superfile
0000006B 2013-02-14 20:44:21 30752 30752 CDistributedFileTransaction: Transaction pausing
0000006C 2013-02-14 20:45:29 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
0000006D 2013-02-14 20:45:34 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000006E 2013-02-14 20:45:34 30752 30752 Backtrace:
0000006F 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000070 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000071 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000072 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000073 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000074 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000075 2013-02-14 20:45:34 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000076 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000077 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000078 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000079 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
0000007A 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
0000007B 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
0000007C 2013-02-14 20:45:34 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
0000007D 2013-02-14 20:45:34 30752 30752   eclagent(main+0x51) [0x400f41]
0000007E 2013-02-14 20:45:34 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
0000007F 2013-02-14 20:45:34 30752 30752   eclagent() [0x401019]
00000080 2013-02-14 20:45:34 30752 30752 CDFAction lock timed out on prachi::superfile
00000081 2013-02-14 20:45:34 30752 30752 CDistributedFileTransaction: Transaction pausing
00000082 2013-02-14 20:47:09 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000083 2013-02-14 20:47:14 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000084 2013-02-14 20:47:14 30752 30752 Backtrace:
00000085 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000086 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000087 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000088 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000089 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000008A 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000008B 2013-02-14 20:47:14 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
0000008C 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
0000008D 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
0000008E 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
0000008F 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000090 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000091 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000092 2013-02-14 20:47:14 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000093 2013-02-14 20:47:14 30752 30752   eclagent(main+0x51) [0x400f41]
00000094 2013-02-14 20:47:14 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000095 2013-02-14 20:47:14 30752 30752   eclagent() [0x401019]
00000096 2013-02-14 20:47:14 30752 30752 CDFAction lock timed out on prachi::superfile
00000097 2013-02-14 20:47:14 30752 30752 CDistributedFileTransaction: Transaction pausing
00000098 2013-02-14 20:48:42 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000099 2013-02-14 20:48:47 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000009A 2013-02-14 20:48:47 30752 30752 Backtrace:
0000009B 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000009C 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000009D 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000009E 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000009F 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000000A0 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000000A1 2013-02-14 20:48:47 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000000A2 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000000A3 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000000A4 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000000A5 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000000A6 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000000A7 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
000000A8 2013-02-14 20:48:47 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
000000A9 2013-02-14 20:48:47 30752 30752   eclagent(main+0x51) [0x400f41]
000000AA 2013-02-14 20:48:47 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
000000AB 2013-02-14 20:48:47 30752 30752   eclagent() [0x401019]
000000AC 2013-02-14 20:48:47 30752 30752 CDFAction lock timed out on prachi::superfile
000000AD 2013-02-14 20:48:47 30752 30752 CDistributedFileTransaction: Transaction pausing
000000AE 2013-02-14 20:49:53 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
000000AF 2013-02-14 20:49:58 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
000000B0 2013-02-14 20:49:58 30752 30752 Backtrace:
000000B1 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
000000B2 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
000000B3 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
000000B4 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
000000B5 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000000B6 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000000B7 2013-02-14 20:49:58 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000000B8 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000000B9 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000000BA 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000000BB 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000000BC 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000000BD 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
000000BE 2013-02-14 20:49:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
000000BF 2013-02-14 20:49:58 30752 30752   eclagent(main+0x51) [0x400f41]
000000C0 2013-02-14 20:49:58 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
000000C1 2013-02-14 20:49:58 30752 30752   eclagent() [0x401019]
000000C2 2013-02-14 20:49:58 30752 30752 CDFAction lock timed out on prachi::superfile
000000C3 2013-02-14 20:49:58 30752 30752 CDistributedFileTransaction: Transaction pausing
000000C4 2013-02-14 20:51:23 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
000000C5 2013-02-14 20:51:28 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
000000C6 2013-02-14 20:51:28 30752 30752 Backtrace:
000000C7 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
000000C8 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
000000C9 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
000000CA 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
000000CB 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000000CC 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000000CD 2013-02-14 20:51:28 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000000CE 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000000CF 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000000D0 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000000D1 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000000D2 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000000D3 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
000000D4 2013-02-14 20:51:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
000000D5 2013-02-14 20:51:28 30752 30752   eclagent(main+0x51) [0x400f41]
000000D6 2013-02-14 20:51:28 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
000000D7 2013-02-14 20:51:28 30752 30752   eclagent() [0x401019]
000000D8 2013-02-14 20:51:28 30752 30752 CDFAction lock timed out on prachi::superfile
000000D9 2013-02-14 20:51:28 30752 30752 CDistributedFileTransaction: Transaction pausing
000000DA 2013-02-14 20:52:58 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
000000DB 2013-02-14 20:53:03 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
000000DC 2013-02-14 20:53:03 30752 30752 Backtrace:
000000DD 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
000000DE 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
000000DF 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
000000E0 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
000000E1 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000000E2 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000000E3 2013-02-14 20:53:03 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000000E4 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000000E5 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000000E6 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000000E7 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000000E8 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000000E9 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
000000EA 2013-02-14 20:53:03 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
000000EB 2013-02-14 20:53:03 30752 30752   eclagent(main+0x51) [0x400f41]
000000EC 2013-02-14 20:53:03 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
000000ED 2013-02-14 20:53:03 30752 30752   eclagent() [0x401019]
000000EE 2013-02-14 20:53:03 30752 30752 CDFAction lock timed out on prachi::superfile
000000EF 2013-02-14 20:53:03 30752 30752 CDistributedFileTransaction: Transaction pausing
000000F0 2013-02-14 20:53:53 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
000000F1 2013-02-14 20:53:58 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
000000F2 2013-02-14 20:53:58 30752 30752 Backtrace:
000000F3 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
000000F4 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
000000F5 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
000000F6 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
000000F7 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000000F8 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000000F9 2013-02-14 20:53:58 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000000FA 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000000FB 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000000FC 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000000FD 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000000FE 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000000FF 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000100 2013-02-14 20:53:58 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000101 2013-02-14 20:53:58 30752 30752   eclagent(main+0x51) [0x400f41]
00000102 2013-02-14 20:53:58 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000103 2013-02-14 20:53:58 30752 30752   eclagent() [0x401019]
00000104 2013-02-14 20:53:58 30752 30752 CDFAction lock timed out on prachi::superfile
00000105 2013-02-14 20:53:58 30752 30752 CDistributedFileTransaction: Transaction pausing
00000106 2013-02-14 20:55:12 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000107 2013-02-14 20:55:18 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000108 2013-02-14 20:55:18 30752 30752 Backtrace:
00000109 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000010A 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000010B 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000010C 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000010D 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000010E 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000010F 2013-02-14 20:55:18 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000110 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000111 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000112 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000113 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000114 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000115 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000116 2013-02-14 20:55:18 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000117 2013-02-14 20:55:18 30752 30752   eclagent(main+0x51) [0x400f41]
00000118 2013-02-14 20:55:18 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000119 2013-02-14 20:55:18 30752 30752   eclagent() [0x401019]
0000011A 2013-02-14 20:55:18 30752 30752 CDFAction lock timed out on prachi::superfile
0000011B 2013-02-14 20:55:18 30752 30752 CDistributedFileTransaction: Transaction pausing
0000011C 2013-02-14 20:56:00 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
0000011D 2013-02-14 20:56:05 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000011E 2013-02-14 20:56:05 30752 30752 Backtrace:
0000011F 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000120 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000121 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000122 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000123 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000124 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000125 2013-02-14 20:56:05 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000126 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000127 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000128 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000129 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
0000012A 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
0000012B 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
0000012C 2013-02-14 20:56:05 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
0000012D 2013-02-14 20:56:05 30752 30752   eclagent(main+0x51) [0x400f41]
0000012E 2013-02-14 20:56:05 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
0000012F 2013-02-14 20:56:05 30752 30752   eclagent() [0x401019]
00000130 2013-02-14 20:56:05 30752 30752 CDFAction lock timed out on prachi::superfile
00000131 2013-02-14 20:56:05 30752 30752 CDistributedFileTransaction: Transaction pausing
00000132 2013-02-14 20:57:21 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000133 2013-02-14 20:57:27 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000134 2013-02-14 20:57:27 30752 30752 Backtrace:
00000135 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000136 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000137 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000138 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000139 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000013A 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000013B 2013-02-14 20:57:27 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
0000013C 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
0000013D 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
0000013E 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
0000013F 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000140 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000141 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000142 2013-02-14 20:57:27 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000143 2013-02-14 20:57:27 30752 30752   eclagent(main+0x51) [0x400f41]
00000144 2013-02-14 20:57:27 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000145 2013-02-14 20:57:27 30752 30752   eclagent() [0x401019]
00000146 2013-02-14 20:57:27 30752 30752 CDFAction lock timed out on prachi::superfile
00000147 2013-02-14 20:57:27 30752 30752 CDistributedFileTransaction: Transaction pausing
00000148 2013-02-14 20:58:23 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000149 2013-02-14 20:58:28 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000014A 2013-02-14 20:58:28 30752 30752 Backtrace:
0000014B 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000014C 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000014D 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000014E 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000014F 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000150 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000151 2013-02-14 20:58:28 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000152 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000153 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000154 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000155 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000156 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000157 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000158 2013-02-14 20:58:28 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000159 2013-02-14 20:58:28 30752 30752   eclagent(main+0x51) [0x400f41]
0000015A 2013-02-14 20:58:28 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
0000015B 2013-02-14 20:58:28 30752 30752   eclagent() [0x401019]
0000015C 2013-02-14 20:58:28 30752 30752 CDFAction lock timed out on prachi::superfile
0000015D 2013-02-14 20:58:28 30752 30752 CDistributedFileTransaction: Transaction pausing
0000015E 2013-02-14 20:59:27 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
0000015F 2013-02-14 20:59:32 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000160 2013-02-14 20:59:32 30752 30752 Backtrace:
00000161 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000162 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000163 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000164 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000165 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000166 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000167 2013-02-14 20:59:32 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000168 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000169 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
0000016A 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
0000016B 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
0000016C 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
0000016D 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
0000016E 2013-02-14 20:59:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
0000016F 2013-02-14 20:59:32 30752 30752   eclagent(main+0x51) [0x400f41]
00000170 2013-02-14 20:59:32 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000171 2013-02-14 20:59:32 30752 30752   eclagent() [0x401019]
00000172 2013-02-14 20:59:32 30752 30752 CDFAction lock timed out on prachi::superfile
00000173 2013-02-14 20:59:32 30752 30752 CDistributedFileTransaction: Transaction pausing
00000174 2013-02-14 21:00:27 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
00000175 2013-02-14 21:00:32 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
00000176 2013-02-14 21:00:32 30752 30752 Backtrace:
00000177 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
00000178 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
00000179 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
0000017A 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
0000017B 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
0000017C 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
0000017D 2013-02-14 21:00:32 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
0000017E 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
0000017F 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000180 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000181 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000182 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000183 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
00000184 2013-02-14 21:00:32 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
00000185 2013-02-14 21:00:32 30752 30752   eclagent(main+0x51) [0x400f41]
00000186 2013-02-14 21:00:32 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
00000187 2013-02-14 21:00:32 30752 30752   eclagent() [0x401019]
00000188 2013-02-14 21:00:32 30752 30752 CDFAction lock timed out on prachi::superfile
00000189 2013-02-14 21:00:32 30752 30752 CDistributedFileTransaction: Transaction pausing
0000018A 2013-02-14 21:01:43 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
0000018B 2013-02-14 21:01:48 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
0000018C 2013-02-14 21:01:48 30752 30752 Backtrace:
0000018D 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
0000018E 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
0000018F 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
00000190 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
00000191 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
00000192 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
00000193 2013-02-14 21:01:48 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
00000194 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
00000195 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
00000196 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
00000197 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
00000198 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
00000199 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
0000019A 2013-02-14 21:01:48 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
0000019B 2013-02-14 21:01:48 30752 30752   eclagent(main+0x51) [0x400f41]
0000019C 2013-02-14 21:01:48 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
0000019D 2013-02-14 21:01:48 30752 30752   eclagent() [0x401019]
0000019E 2013-02-14 21:01:48 30752 30752 CDFAction lock timed out on prachi::superfile
0000019F 2013-02-14 21:01:48 30752 30752 CDistributedFileTransaction: Transaction pausing
000001A0 2013-02-14 21:02:45 30752 30752 safeChangeModeWrite - temporarily releasing lock on prachi::superfile to avoid deadlock
000001A1 2013-02-14 21:02:50 30752 30752 safeChangeModeWrite on prachi::superfile waiting for 10s
000001A2 2013-02-14 21:02:50 30752 30752 Backtrace:
000001A3 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x7f524483c4f6]
000001A4 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_Z19safeChangeModeWriteP17IRemoteConnectionPKcRbj+0x1ff) [0x7f52437d229f]
000001A5 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN20CDistributedFileBaseI21IDistributedSuperFileE14lockPropertiesEj+0x64) [0x7f52437680c4]
000001A6 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN21CDistributedSuperFile17cAddSubFileAction7prepareEv+0xcb) [0x7f524376ff9b]
000001A7 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libdalibase.so(_ZN27CDistributedFileTransaction6commitEv+0x4e) [0x7f5243776a3e]
000001A8 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/plugins/libfileservices.so(fslFinishSuperFileTransaction+0x4f) [0x7f5236510d7f]
000001A9 2013-02-14 21:02:50 30752 30752   /var/lib/HPCCSystems/myeclccserver/libW20130214-203956.so(+0xb22e) [0x7f5235f8e22e]
000001AA 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11performItemEjj+0x54) [0x7f5243044504]
000001AB 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine13doExecuteItemER20IRuntimeWorkflowItemj+0x41) [0x7f5243045121]
000001AC 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine11executeItemEjj+0x2b7) [0x7f5243044d17]
000001AD 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libeclrtl.so(_ZN15WorkflowMachine7performEP18IGlobalCodeContextP11IEclProcess+0x17c) [0x7f52430454cc]
000001AE 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent10runProcessEP11IEclProcess+0x147) [0x7f5244c6c977]
000001AF 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_ZN8EclAgent9doProcessEv+0x277) [0x7f5244c6ee97]
000001B0 2013-02-14 21:02:50 30752 30752   /opt/HPCCSystems/lib/libhthor.so(_Z13eclagent_mainiPPKcP12StringBufferb+0x7a0) [0x7f5244c719c0]
000001B1 2013-02-14 21:02:50 30752 30752   eclagent(main+0x51) [0x400f41]
000001B2 2013-02-14 21:02:50 30752 30752   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f5243ec576d]
000001B3 2013-02-14 21:02:50 30752 30752   eclagent() [0x401019]
000001B4 2013-02-14 21:02:50 30752 30752 CDFAction lock timed out on prachi::superfile
000001B5 2013-02-14 21:02:50 30752 30752 CDistributedFileTransaction: Transaction pausing

I am attaching images of ECL Watch and ECL IDE.

Thanks and Regards!
Attachments
img3.PNG
Logical file on cluster details.
(65.51 KiB) Downloaded 2005 times
img2.PNG
Workunit details
(53.45 KiB) Downloaded 2005 times
img1.PNG
ECL IDE screenshot highlighting workunit as running.
(91.42 KiB) Downloaded 2005 times
prachi
 
Posts: 46
Joined: Mon Jul 23, 2012 11:50 am

Sat Feb 16, 2013 3:49 pm Change Time Zone

Hi,

As described above i was working on Superfile.I also tried working with simple logical file.Steps were:
THOR code:
1.Create one logical file record structure.
2.Build INDEX on it.

ROXIE code:
1.Publish code to access this logical file on ROXIE.

And then again run THOR code.

BUT when i run THOR code 2nd time, workunit goes in running state for indefinite time ! And my other workunit (other than this ECL code) goes in blocked state.

My assumption is: may be after publishing on ROXIE, if i try to overwrite those logical file with target cluster as THOR then this problem is getting created.

So i tried above scenario again but with little change in it.THIS time i deleted published query on ROXIE and then submitted THOR code for second time and it completed successfully!!

BUT BUT after overwriting logical file when i tried to PUBLISH query on ROXIE (to access this logical file), it FAILED!

Questions:
1.So is it that we cant overwrite logical file with target cluster as THOR and which is copied on ROXIE cluster as well???

2.Why after overwriting logical file on THOR it is not getting PUBLISHED again???

Waiting for reply/response.

Thanks and Regards!
prachi
 
Posts: 46
Joined: Mon Jul 23, 2012 11:50 am

Sat Feb 16, 2013 4:36 pm Change Time Zone

I think there are two general 'truths' to keep in mind that may help you craft a solution:

* Published Roxie queries acquire locks on logical files, indexes, and superfiles/superkeys. This happens when the query is published, and released when the query is unpublished.

* When any of those file structures are locked, you cannot modify them (e.g. add a subkey to a superkey). Attempting to modify them will result in the behavior you see (the update process waiting a long time to acquire its own lock, then eventually giving up and failing).

Therefore, in order to update the data that is being used by a published Roxie query, you will need to first unpublish the query (actually, all queries that reference the data), perform the update, then republish the query/queries. That is the basis for performing an update. Exactly how you perform these steps depend on your external requirements, your HPCC environment (e.g. a multi-cluster configuration behind a load balancer offers more flexibility) and your query-to-data configurations.

You may want to search the forum more, as this topic has been discussed numerous times. One possible solution may be within http://hpccsystems.com/bb/viewtopic.php?f=8&t=550&p=2567 but I'm given to understand that it may not work in all situations/environments. It may give you some ideas, though.

Hope this helps.

Dan
DSC
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 568
Joined: Tue Oct 18, 2011 4:45 pm

Sun Feb 17, 2013 7:12 am Change Time Zone

Hi Dan,

Thanks for your input.

I need to build a workaround in which unpublish and publish of ROXIE query fits in.

I also have the same doubt which u had.
1.As described above, web application users will query on ROXIE (say clientdata.1).If at that time i am unpublishing that query (clientdata.1) then user will face an error. How to handle this situation then??

2.How to determine exact query name?

Thanks and Regards!
prachi
 
Posts: 46
Joined: Mon Jul 23, 2012 11:50 am

Tue May 28, 2013 1:19 pm Change Time Zone

If you want to update the contents of superkeys used by roxie without removing queries from roxie, here is what I recommend.

. Have multiple dalis. 1 for Thor and 1 for roxie. Roxie will lock the files in the thor dali while roxie is accessing the file remotely -i.e. when copying the file. Once roxie has the files local, it should remove the remote lock.

. Use packages to define the contents of superkeys (use latest 3.10.x release on the portal). This will cause roxie to lock the subkeys, but not the superkeys

. Remember as you add more and more subkeys to a superkey, you may decrease performance. Consolidate data from time to time

The "hang" when roxie has a lock on it should now be fixed. Users should get a message indicating the action could not be completed, which is better than hanging with no information.
sort
 
Posts: 59
Joined: Wed May 11, 2011 1:54 pm


Return to ECL IDE

Who is online

Users browsing this forum: No registered users and 1 guest

cron