Spraying Word document in HPCC THOR

[rajesh.dorairaj]

Dear Team,

We are looking for below business case.
- Upload/Spray multiple Word documents(Unstructured data) in distributed system
- Provide UI to achieve below features
- Searching on a word or Sentence should return the matching section/paragraph from the sprayed document

My idea is to use HPCC. As far as my knowledge on HPCC, I can spray and build index of the uploaded data in THOR and build a Query in Roxie to achieve Search functionality.
Please confirm on my above understanding.

My question is on whether HPCC will support uploading/spraying word document(s).

[sameermsc]

Did a quick test on my machine, Even though HPCC does not complain while uploading and spraying word documents, the populated dataset (after reading the sprayed file) contains lot of unnecessary/Junk data that you dont want.

If possible can you convert word documents into a txt file and work on them

Regards,
Sameer

[tlhumphrey2]

Before you upload Word documents, first convert them to plain text (Word can do that for you).

Or, on the web, you can find C++ code that converts Word documents to plain text. And, you can make a ECL function for this code, by wrapping it with BEGINC++ and ENDC++ (please read the ECL Language Reference for the details).

[faarisuman]

I have about 10 files that are related by a 3 field composite key (roughly 45 bytes total). In the RDBMS world I would be inclined to convert the natural key into a numeric surrogate key to reduce the footprint and hopefully improve sorts and joins. The 3 key fields constitute 20-30% of the total data size.


______________
You can easily check out our high quality itil which prepares you well for the ccent questions You can also get success in real Test-king Certification exam with the quality www.quincy.edu and University of Saint Joseph and best of luck.

[rtaylor]

faari,

That sounds like a reasonable approach to use in HPCC, too.

HTH,

Richard