Fri Nov 16, 2018 2:10 pm
Login Register Lost Password? Contact Us


How to dynamically import data and model data?

Comments or questions on structuring and organizing your data

Tue Jul 29, 2014 6:18 pm Change Time Zone

Hi all,

I was wondering if there was a way to dynamically import data to Thor, since I am dealing with some real-time data. Thank you very much.

Sincerely,
James
rqg0717
 
Posts: 27
Joined: Fri Jul 25, 2014 6:00 pm

Tue Jul 29, 2014 8:03 pm Change Time Zone

Hi James,

There are three ways to import data into THOR:

1. Using the ECL Watch interface
2. Using the DFUPLUS Command Line Utility
3. Using the ECL Standard Function Libraries

So to dynamically import data into THOR, you probably want to use option 3. This is a standard operation that we do everyday, as we are always getting updates, some daily, some weekly, some monthly. The trick is to come up with a standard to detect that new data is on the landing zone and ready for import. One technique is to use a CRON job to schedule an ECL event that is triggered by the existence of a "trigger" or "dummy" file that signals the CRON job that data is ready for import. When that file is detected, the CRON job fires an event that starts the dynamic spray process.

Look at CRON and WHEN and EVENT in the Language Reference Manual, as well as the Spray Libraries that can be found in the Standard Library Reference PDF.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 975
Joined: Wed Jun 29, 2011 7:13 pm

Tue Jul 29, 2014 8:06 pm Change Time Zone

Hi Bob,

Wow, you are really an expert. You answered all the questions I have asked recently. I will look into the option 3. Thanks a lot.

bforeman wrote:Hi James,

There are three ways to import data into THOR:

1. Using the ECL Watch interface
2. Using the DFUPLUS Command Line Utility
3. Using the ECL Standard Function Libraries

So to dynamically import data into THOR, you probably want to use option 3. This is a standard operation that we do everyday, as we are always getting updates, some daily, some weekly, some monthly. The trick is to come up with a standard to detect that new data is on the landing zone and ready for import. One technique is to use a CRON job to schedule an ECL event that is triggered by the existence of a "trigger" or "dummy" file that signals the CRON job that data is ready for import. When that file is detected, the CRON job fires an event that starts the dynamic spray process.

Look at CRON and WHEN and EVENT in the Language Reference Manual, as well as the Spray Libraries that can be found in the Standard Library Reference PDF.

Regards,

Bob
rqg0717
 
Posts: 27
Joined: Fri Jul 25, 2014 6:00 pm

Tue Jul 29, 2014 8:10 pm Change Time Zone

Thanks James, I have been working with and teaching ECL now for over 3 1/2 years, and am blessed to be surrounded by a team of very talented individuals. Welcome to the community, and always glad to help!

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 975
Joined: Wed Jun 29, 2011 7:13 pm


Return to Data Modeling

Who is online

Users browsing this forum: No registered users and 1 guest