Sun Aug 14, 2022 4:26 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl ( This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.

Record definition for a sprayed file

Topics around the use of the JDBC driver for HPCC Systems

Tue Jan 29, 2013 1:51 pm Change Time Zone


I have attached a file that has the record in my csv(as csv file upload is not allowed in the forum)I have sprayed onto a HPCC cluster using ECLWatch.

Logical Name: .::adcampaign_twitter_data

I'm using HPCC JDBC Driver to access the sprayed data. The query is :

Code: Select all
String stmtsql = "select tbl.* from .::adcampaign_twitter_data tbl where User_ID='294766'";

The error that I'm getting is as follows:

Code: Select all
2013/01/29 07:05:28.664 - HPCCStatementAttempting to process sql query: select tbl.* from .::adcampaign_twitter_data tbl where User_ID='294766'
2013/01/29 07:05:28.664 - INCOMING SQL: select tbl.* from .::adcampaign_twitter_data tbl where User_ID='294766'
2013/01/29 07:05:28.679 - Fetching file information: Files and Superfiles
2013/01/29 07:05:28.726 - .::adcampaign_twitter_data does not appear to contain a valid record definition.
java.sql.SQLException: HPCCStatement:  Cannot execute SQL command
   Invalid or forbidden table found: .::ADCAMPAIGN_TWITTER_DATA
   at org.hpccsystems.jdbcdriver.HPCCStatement.executeHPCCQuery(
   at org.hpccsystems.jdbcdriver.HPCCStatement.execute(
   at org.hpccsystems.jdbcdriver.HPCCStatement.execute(
   at org.hpccsystems.jdbcdriver.tests.HPCCDriverTest.main(
2013/01/29 07:05:28.726 - Invalid or forbidden table found: .::ADCAMPAIGN_TWITTER_DATA

As per the HPCC JDBC documentation :

Note: The HPCC JDBC driver only supports files which contain the record definition in the logical file’s
metadata. Sprayed files do not contain this metadata. This metadata exists on any file or index which is
written to the HPCC Distributed File System. Sprayed data files typically undergo some processing and an
OUTPUT of the transformed data to disk before use, so this should not interfere with the driver’s usefulness.

In my case, how must I upload the file onto HPCC - I used the word upload because sprayed files don't seem to be supported by HPCC-JDBC ! Shall I use ECL Plus, DFU etc. for upload?Or something else?

Thanks and regards !
Logical Name: .::adcampaign_twitter_data
(1.1 KiB) Downloaded 858 times
Posts: 43
Joined: Mon Jul 23, 2012 11:23 am

Tue Jan 29, 2013 3:42 pm Change Time Zone

Metadata about a dataset is maintained by the DFU and gets written when the OUTPUT action writes the file to disk. To make the sprayed file available to the JDBC driver, just define the sprayed file then write it to disk (with a new filename) with OUTPUT -- this will give the JDBC driver the metadata it needs.


Community Advisory Board Member
Community Advisory Board Member
Posts: 1619
Joined: Wed Oct 26, 2011 7:40 pm

Tue Jan 29, 2013 5:06 pm Change Time Zone

Kaliyugantagonist, as Richard pointed out, adding a record definition to a file involves defining the data structure, loading the data into an HPCC dataset, and OUTPUTing the dataset out to an HPCC file...

something like this:

twitter_data_structure :=RECORD
type fieldname1;
type fieldname2;
type fieldnameN;

twitter_data = DATASET('.::adcampaign_twitter_data', twitter_data_structure, CSV);
OUTPUT(twitter_data, , '.::hpcc::adcampaign_twitter_data');

now you should be able to query '.::hpcc::adcampaign_twitter_data' using SQL through the jdbc driver. Let us know if there are any questions/concerns/comments. Thanks.
[email protected]
Posts: 22
Joined: Fri Oct 07, 2011 2:24 pm

Wed Feb 06, 2013 9:26 am Change Time Zone

Hello everyone,

Thanks for the inputs - they helped a lot !
Please treat this question as closed.

Thanks and regards !
Posts: 43
Joined: Mon Jul 23, 2012 11:23 am

Return to Java Tools

Who is online

Users browsing this forum: No registered users and 1 guest