Configuring multi-node HPCC System in AWS

[Anjali]

Hi,

As part of study i was asked to set up a multi-node system on Amazon instances,in such a way that instance1 will be of AWS account1 and instance2 will be of AWS account2.

Is it possible to configure multi-node system on AWS instances that are of different account(here between instance1 and instance2)?


Can anyone provide a clarification?


Thanks,
Anjali

[rtaylor]

Anjali,

As part of study i was asked to set up a multi-node system on Amazon instances,in such a way that instance1 will be of AWS account1 and instance2 will be of AWS account2.

So you have two user accounts on AWS. You want to ramp up a single HPCC environment where the nodes are billed to two separate AWS accounts? How do you expect to do that? The nodes must talk to each other to work together.

So, my real question is: Why?

Richard

[Anjali]

Hi,

Thanks for your reply

Yes,that is the scenario i have given.We are a study group with our own AWS free accounts.Just for knowledge purpose we are trying to set up a cluster between our free instances.

So is it possible to configure the multi node system between instances of different account?is it possible to have a connect between these free instances?

Thanks,
Anjali

[tlhumphrey2]

It is possible to have an HPCC System whose nodes are on different accounts. The setup process doesn't ask any information about the AWS accounts. I do have one caveat. If you want the HPCC System to be fast, the instances should be in the same AWS Placement Group, but it isn't a requirement. I don't know if you can have the same placement group in two accounts -- probably not.

With that said, the key to making all this work is getting the programs that setup the HPCC System on all instances and making sure that all instances have the correct ports setup in their security group. You can get all the programs for setting up the HPCC System on github at https://github.com/tlhumphrey2/EasyFastHPCCoAWS.

The document in this github repository, EasyFastHPCCoAWS.pdf, tells you how to setup an HPCC System using the programs and the CloudFormation template in the repository. You won't be able to use CloudFormation since your HPCC System will be made instances from different accounts. But, you can use the programs.

[tlhumphrey2]

Another alternative.
You can have multiple HPCC nodes on the same instance. And, if the instance type is one with multiple cores, lots of memory and disk space, the HPCC System can run fairly fast.

[Anjali]

Hi,

Thank you for your valuable comments..

I hope the git hub link that you have shared can help me to sort out this.Let me try once:)

Thanks,
Anjali

[tlhumphrey2]

The following document is the best place to start: http://cdn.hpccsystems.com/pdf/RunningHPCCwithinAWS_EC2_Beta1.pdf.

It gives the basics. The github repository I sited is more advanced and therefore leaves out details you need to know.

[Anjali]

Sure...Thanks a lot