Can we spray binary data on multiple nodes

[Ankita Singla]

Hi..

As if we spray CSV, XML or FLAT files, then distribution of the file across all the nodes of the target cluster is such that each single record is always whole and complete on a single node and the records are "evenly" distributed across the nodes.

But in binary data my one record can be on 50 rows and other can be on 2 rows. Also each row can differ in columns then how spraying works...?????
Is there any option to define delimiters so that it know different records and spray it on different nodes...

Plzz help..

[rtaylor]

Ankita,

But in binary data my one record can be on 50 rows and other can be on 2 rows. Also each row can differ in columns then how spraying works...?????

We work with binary data files all the time, with both fixed-length and variable-length records. Typically, each record has the same structure and all the data is contained in that one record with a delimiter indicating the end of the record. Spraying this type of data simply requires knowing what the record delimiter is.

The closest thing that I can think of to what I think you're describing would be a file wherein the data (either binary or textual) is comprised of multiple rows of different record types which are grouped together to describe a single logical entity in your data. These multiple physical records would be grouped either by a grouping identifier in the data (the logical entity identifier), or simply the physical positioning of the records in the file -- such as:

• record type 10 is always first,
• always followed by a 20,
• then optionally some 30s and/or 40s,
• and the next grouping begins with the next instance of a 10

If this is the kind of data you're working with, then spraying is not a problem as long as each record type has the same delimniter, because it doesn't matter whether the logical grouping is spread across nodes, only that each physical record is whole and complete on a single node. The spray maintains the physical order of records in the file that is sprayed, so that the first n number of records are put on node 1, then the first record on node 2 is n+1, etc.

Your real problem with this kind of data is in defining the RECORD structure for the file so that you can easily work with it. Take a look at the IFBLOCK discussion in the RECORD structure documentation for one way to accomplish this.

However, if I've misunderstood and this is not the type of data you're working with, then can you please expand on your description so we can better help you?

HTH,

Richard

[Ankita Singla]

Thanks for the reply.....
May be it can help me in some way...