Mon Mar 19, 2018 6:33 am
Login Register Lost Password? Contact Us

Spraying from S3

Questions or comments related to Cloud Computing and the HPCC Systems Instant Cloud for AWS

Thu May 09, 2013 4:20 am Change Time Zone

Is it possible to spray data directly from s3
also my files are in gz format is it possible to spray them directly without unzipping them?
Community Advisory Board Member
Community Advisory Board Member
Posts: 105
Joined: Mon Oct 17, 2011 6:48 pm

Thu May 09, 2013 6:20 pm Change Time Zone

There are a few ways you can spray from S3 but you can't spray directly from S3. The easiest is to use one of the amazon utilities such as s3fs which will allow you to mount your s3 bucket as a drive and use it as the landing zone. Depending on your file count and size this is usually a decent solution. In many cases its quicker to copy the file over and work with it than using the s3 bucket as a mounted drive though.

I'm checking into the gz question, there has been talk about implementing this but I haven't tried it on the newest release.
Posts: 21
Joined: Wed Apr 27, 2011 1:07 pm

Thu May 09, 2013 6:36 pm Change Time Zone

You have to unzip it first. You can upload it and unzip with ECL pipe.

Here is an example of how to unzip it with ECL pipe:
Code: Select all
string filename := 'stocks_20130417.csv.gz' : STORED('filename');
boolean gunzip := true : STORED('gunzip');
boolean bzip := false : STORED('bzip');

rTest := record
string1000 s;

dTest := PIPE(MAP(gunzip=>'gunzip', 'bzip2 -d ') + ' /var/lib/HPCCSystems/dropzone/' + filename, rTest, csv);
Posts: 21
Joined: Wed Apr 27, 2011 1:07 pm

Return to Cloud

Who is online

Users browsing this forum: No registered users and 1 guest