Sun Aug 14, 2022 3:58 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl (https://stackoverflow.com/search?tab=newest&q=hpcc-ecl). This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.



Spraying a file takes a long time to initiate on AWS

Questions or comments related to Cloud Computing and the HPCC Systems Instant Cloud for AWS

Fri Oct 19, 2012 12:23 pm Change Time Zone

Spraying an 8GB file on a 10 node cluster on AWS takes a total of 9 minutes+ but there is a long wait for almost 7 minutes where it stays at 0%. (The actual spraying finishes within 2 minutes once it starts)

Code: Select all
Time Taken   :   9m 21s
KB Per Second   :   82618
KB Per Second Average   :   82618
Progress Message   :   100% Done, 0 secs left (987/987MB @82618KB/s) current rate=82618KB/s [10/10nodes]
Summary Message   :   Total time taken 9m 21s, Average transfer 82618Kb/sec


1. Are there ways of speeding up the initial wait where it stays at 0%?
2. Are there any logs we could 'tail' to observe the activity throughout the spraying process?

Regards
Arun
arunarav
 
Posts: 20
Joined: Fri Sep 21, 2012 5:34 pm

Mon Oct 22, 2012 8:06 pm Change Time Zone

Something doesn't add up with your stats displayed (could it be a typo?)

If it's an 8GB file then :
@82618KB/s , is not ~ 9m12s ( more like 96s )

Do you know if the data is being pulled or pushed? I assume pulled since I think that is the default.

There should be log files – e.g., ftslave.log (one for each target node). That should tell you when the process was started to read the data and some progress information. There should be a master log (in the dali log?) that traces what happens on the master node.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1006
Joined: Wed Jun 29, 2011 7:13 pm


Return to Cloud

Who is online

Users browsing this forum: No registered users and 1 guest

cron