Data Ingestion - Automation
Can I automate data ingestion to Thor cluster and move the ETL-ed data to Roxie?
For example, I want to direct the flume (Apached Flume) sources to HPCC (considering HPCC thor as sink). Then automate the ETL process with my ECL queries and refined data to be published to Roxie.
With this setup, it is possible to get near real time analytics with HPCC engine, any thoughts or references will help. Thanks in Advance.
Regards
Durai
For example, I want to direct the flume (Apached Flume) sources to HPCC (considering HPCC thor as sink). Then automate the ETL process with my ECL queries and refined data to be published to Roxie.
With this setup, it is possible to get near real time analytics with HPCC engine, any thoughts or references will help. Thanks in Advance.
Regards
Durai
- Durai
- Community Advisory Board Member
- Posts: 24
- Joined: Thu Jun 23, 2011 2:50 pm
Durai,
You can automate data file sprays to Thor using either the DFUplus.exe command line utility or the STD.File.Spray... functions.
Then you can automate the subsequent ETL processing of that data on Thor using the ECLplus.exe command line utility.
Finally, you can automate Publishing to Roxie using the ECL.exe command line utility.
These command line utilities are all documented in the ClientTools.PDF available for download here: http://hpccsystems.com/community/docs/e ... leinttools
HTH,
Richard
You can automate data file sprays to Thor using either the DFUplus.exe command line utility or the STD.File.Spray... functions.
Then you can automate the subsequent ETL processing of that data on Thor using the ECLplus.exe command line utility.
Finally, you can automate Publishing to Roxie using the ECL.exe command line utility.
These command line utilities are all documented in the ClientTools.PDF available for download here: http://hpccsystems.com/community/docs/e ... leinttools
HTH,
Richard
- rtaylor
- Community Advisory Board Member
- Posts: 1619
- Joined: Wed Oct 26, 2011 7:40 pm
Thanks Richard. I am looking into the details now.
- Durai
- Community Advisory Board Member
- Posts: 24
- Joined: Thu Jun 23, 2011 2:50 pm
3 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 1 guest