Data Ingestion - Automation

Mon Apr 09, 2012 4:10 am

Can I automate data ingestion to Thor cluster and move the ETL-ed data to Roxie?

For example, I want to direct the flume (Apached Flume) sources to HPCC (considering HPCC thor as sink). Then automate the ETL process with my ECL queries and refined data to be published to Roxie.

With this setup, it is possible to get near real time analytics with HPCC engine, any thoughts or references will help. Thanks in Advance.

Regards
Durai

Mon Apr 09, 2012 7:27 pm

Durai,

You can automate data file sprays to Thor using either the DFUplus.exe command line utility or the STD.File.Spray... functions.

Then you can automate the subsequent ETL processing of that data on Thor using the ECLplus.exe command line utility.

Finally, you can automate Publishing to Roxie using the ECL.exe command line utility.

These command line utilities are all documented in the ClientTools.PDF available for download here: http://hpccsystems.com/community/docs/e ... leinttools

HTH,

Richard

Tue Apr 10, 2012 4:16 am

Thanks Richard. I am looking into the details now.