Sat Aug 18, 2018 11:41 pm
Login Register Lost Password? Contact Us


Data Ingestion - Automation

Post questions or comments on how best to manage your big data problem

Mon Apr 09, 2012 4:10 am Change Time Zone

Can I automate data ingestion to Thor cluster and move the ETL-ed data to Roxie?

For example, I want to direct the flume (Apached Flume) sources to HPCC (considering HPCC thor as sink). Then automate the ETL process with my ECL queries and refined data to be published to Roxie.

With this setup, it is possible to get near real time analytics with HPCC engine, any thoughts or references will help. Thanks in Advance.

Regards
Durai
Durai
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 24
Joined: Thu Jun 23, 2011 2:50 pm

Mon Apr 09, 2012 7:27 pm Change Time Zone

Durai,

You can automate data file sprays to Thor using either the DFUplus.exe command line utility or the STD.File.Spray... functions.

Then you can automate the subsequent ETL processing of that data on Thor using the ECLplus.exe command line utility.

Finally, you can automate Publishing to Roxie using the ECL.exe command line utility.

These command line utilities are all documented in the ClientTools.PDF available for download here: http://hpccsystems.com/community/docs/e ... leinttools

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1370
Joined: Wed Oct 26, 2011 7:40 pm

Tue Apr 10, 2012 4:16 am Change Time Zone

Thanks Richard. I am looking into the details now.
Durai
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 24
Joined: Thu Jun 23, 2011 2:50 pm


Return to Managing Big Data

Who is online

Users browsing this forum: No registered users and 1 guest

cron