Sun Aug 14, 2022 4:11 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl (https://stackoverflow.com/search?tab=newest&q=hpcc-ecl). This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.



Thor on Amazon with on-premise Roxie

Questions or comments related to Cloud Computing and the HPCC Systems Instant Cloud for AWS

Tue Oct 09, 2012 12:40 pm Change Time Zone

We would like to create some proof-of-concepts on varying dataset size (ranging from 300 GB to 10 TB).

In order to get a quick ramp-up, we are looking to leverage the cloud. My understanding is that Thor is available on Amazon AWS but Roxie is not yet available.

Request some insight into the following questions:

(1) Is this a feasible option (ie Thor on Amazon with on-premise Roxie?)

(2) What are the suggestions around using Thor on Amazon with on-premise Roxie? What network and infrastructure would be required to demonstrate ETL on Thor with low latency querying on Roxie?

(3) I read on the blog that a thumb rule for estimating Thor disk space is usually about 3 times the size of the aggregated input data. How should we go about estimating the # of Roxie nodes?

Thanks
arunarav
 
Posts: 20
Joined: Fri Sep 21, 2012 5:34 pm

Tue Oct 09, 2012 1:04 pm Change Time Zone

I just did a quick check, and it looks like Roxie nodes are now available on the one-click THOR AWS cluster. That would be your best bet.

The rule of thumb for the ratio of THOR to ROXIE clusters is that it be evenly divisible. Typically in a 400-node production cluster you will see a 100 node Roxie.

Hope this helps,

Bob
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1006
Joined: Wed Jun 29, 2011 7:13 pm

Wed Oct 10, 2012 3:26 pm Change Time Zone

Bob: Thanks for the update. Will there be an equivalent of one-click Thor for Roxie? Could you kindly point me to the documentation to get Roxie instances going on AWS?

Thanks
-Arun
arunarav
 
Posts: 20
Joined: Fri Sep 21, 2012 5:34 pm

Wed Oct 10, 2012 3:34 pm Change Time Zone

Once you log into AWS and then Launch Cluster, you should see the entry field for adding any number of Roxie nodes that you need. On the one-click interface they are created together. Rule of thumb is that your THOR nodes should be a multiple of your Roxie nodes, for example if you create 10 THOR nodes, 5 Roxie nodes would be a good choice, and adding 2 support nodes you come up with 17 nodes total created by the one-click.
bforeman
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1006
Joined: Wed Jun 29, 2011 7:13 pm


Return to Cloud

Who is online

Users browsing this forum: No registered users and 1 guest

cron