Tue Oct 26, 2021 12:31 pm
Login Register Lost Password? Contact Us


Can Roxie work with hadoop?

Topics related to the Hadoop Connector or migrating data from Hadoop

Thu Jul 14, 2011 1:57 pm Change Time Zone

Hi,

I am just new to HPCC. Currently, we are running hadoop for data process and storage and like what hadoop can do so far. For query part, we are wondering whether we can put Roxie in front of hadoop for searching. Is it possible and easy?

Thanks,
hli
 
Posts: 21
Joined: Thu Jul 14, 2011 1:52 pm

Thu Jul 14, 2011 2:29 pm Change Time Zone

Hey There,

Roxie works from keys (or indexes) that are built for it by Thor. It is theoretically possible for someone to build roxie keys from Hadoop but it would be a substantial undertaking.

What I would recommend would be installing THOR on the nodes used for hadoop (the systems can co-reside). Do any processing you wish in hadoop; then use a very short THOR process to build the roxie keys. Then you would be able to use Roxie as your search engine and Hadoop for the 'bulk of' your batch work.

At the moment you would want the 'end' of your hadoop process to write out regular Linux files (perhaps in CSV) to allow them to be directly read in THOR. There is a rather more automated HDFS->THOR module in the works: currently slated for Q4

http://hpccsystems.com/products-and-services/products/modules/hadoop-to-roxie-data-export
dabayliss
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 109
Joined: Fri Apr 29, 2011 1:35 pm


Return to From Hadoop

Who is online

Users browsing this forum: No registered users and 1 guest