Wed Aug 21, 2019 7:25 am
Login Register Lost Password? Contact Us


HPCC as a back-end???

Comments and questions related to the Enterprise Control Language

Tue Jan 29, 2013 4:16 am Change Time Zone

I'm a HPCC beginner.

I sprayed multiple csv files using ECL and DFU and also have fetched the data using ECL record structure. I read that XML files can also be the input and that there is a JDBC driver too which supports read-only queries to the HPCC data.

In my Java-based web application, I'm currently using a MySQL database. It consists of several tables which have BLOB columns but the data arriving is huge and we have decided to switch to a HPCC for quick storage and retrieval.

My questions are as follows:

    How do insert my data directly into HPCC - I don't have any kind of file with me(csv, flat, xml etc.)?
    The BLOBs columns in the tables are the result of certain external operations which finish after the initial insert i.e the BLOBs are later 'updated' into the tables - how do I handle such a scenario in HPCC(ECL approach?)
kaliyugantagonist
 
Posts: 43
Joined: Mon Jul 23, 2012 11:23 am

Tue Jan 29, 2013 1:32 pm Change Time Zone

Hi,

First, let me clarify the term "CSV" - You would normally think of CSV as an Excel file, but it can really be any variable length text, you can even treat a whole document as a single record, just specify no seperators, no quotes and go.

If I remember correctly, there is a great article in the Programmer's Guide regarding processing BLOBs in HPCC. Check it out:

http://cdn.hpccsystems.com/releases/CE-Candidate-3.10.0/docs/ECLProgrammersGuide-3.10.0-1.pdf

See Page 36.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 984
Joined: Wed Jun 29, 2011 7:13 pm

Tue Jan 29, 2013 1:41 pm Change Time Zone

Hi bforeman,

I had gone through the PDF that you have suggested.

I'm currently stuck with the two questions that I have posted.

Thanks and regards !
kaliyugantagonist
 
Posts: 43
Joined: Mon Jul 23, 2012 11:23 am

Tue Jan 29, 2013 1:58 pm Change Time Zone

How do insert my data directly into HPCC - I don't have any kind of file with me(csv, flat, xml etc.)?


All data moved directly into HPCC need to be sprayed, and a file needs to be on a landing zone. That file can be exported from MySQL, or perhaps you can use the JDBC driver. I will discuss your option with my colleagues.


Code: Select all
The BLOBs columns in the tables are the result of certain external operations which finish after the initial insert i.e the BLOBs are later 'updated' into the tables - how do I handle such a scenario in HPCC(ECL approach?)


Well, of course, updates are possible on HPCC :-) Perhaps you need to explore the superfiles technique, where files that are alike can be updated when needed. A Superfile is a single logical file that contains many "sub-files" of the exact same layout, and using the superfiles libraries you can add and update as needed.

Regards,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 984
Joined: Wed Jun 29, 2011 7:13 pm

Tue Jan 29, 2013 2:12 pm Change Time Zone

Hi Bob,

I will check out the superfiles approach.

Actually, I had thought about the HPCC-JDBC driver and working on it but I'm stuck - would be glad if you help http://hpccsystems.com/bb/viewtopic.php?f=34&t=719&sid=d743978e56d5a8ed4482dc2adee50081

Thanks and regards !
kaliyugantagonist
 
Posts: 43
Joined: Mon Jul 23, 2012 11:23 am


Return to ECL

Who is online

Users browsing this forum: No registered users and 0 guests

cron