Thu Jun 30, 2022 6:40 pm
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl (https://stackoverflow.com/search?tab=newest&q=hpcc-ecl). This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.



Realtime insert/update operation with JDBC driver on roxie

Topics around the use of the JDBC driver for HPCC Systems

Fri Apr 22, 2022 9:27 am Change Time Zone

Hi Team,

After research I came to know that HPCC can process million of data in real time with less time. I am trying to evaluate a process to do real time data insert or modify with jdbc driver in java. But no luck to crack for real time data writing using jdbc driver. I did it for read/select operation and it works.

Can any one please help here to workout.

We are working with JDBC driver to fetch data from logical file in java.

Below are snap from our code-

String sqlQuery ="select * from tutorial::yn::transactions";
org.hpccsystems.ws.client.utils.Connection connection = new Connection("http://localhost:8510");
HPCCWsSQLClient mywssqlclient = HPCCWsSQLClient.get(connection);
System.out.println(mywssqlclient.getResults(mywssqlclient.executeSQLWUIDResponse(sqlQuery, "roxie", ""), 0, 50));


Till now we are ok with fetching data from file but we have real time requirement of INSERT/UPDATE operations on existing data of file.

Can someone please suggest us that we can support realtime INSERT/UPDATE operation on logical file?
If yes then how can we fulfill our requirements?
adityac
 
Posts: 2
Joined: Thu Apr 21, 2022 6:59 am

Fri May 06, 2022 2:55 pm Change Time Zone

adityac,
Can someone please suggest us that we can support realtime INSERT/UPDATE operation on logical file?
The first thing to understand about the HPCC Systems platform is that it is NOT an RDBMS and not originally designed for OLTP use. It is a Big Data processing platform that was created to do batch processing of huge amounts of data (too big for effective use on any RDBMS). The JDBC driver is designed to allow external access, only, and does not support write operations.
If yes then how can we fulfill our requirements?
It IS possible to make an HPCC Systems platform emulate an OLTP system by adding in a "Deltabase" -- a small SQL database that contains all added/changed records until they can be rolled into the ROXIE data.

An intern in our Brazil office did a Deltabase project and submitted this to our 2020 Community Day conference Poster Competition: https://wiki.hpccsystems.com/display/hpcc/Johny+Chen+Jy+-+2020+Poster+Contest+Resources

And Fujio Turner did a TechTalk presentation in 2017 that also discussed this technique: https://hpccsystems.com/blog/download-tech-talks-hpcc-systems-community-episode-2

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1619
Joined: Wed Oct 26, 2011 7:40 pm


Return to Java Tools

Who is online

Users browsing this forum: No registered users and 1 guest

cron