Sun Jul 22, 2018 4:12 pm
Login Register Lost Password? Contact Us


CLI to manage cluster

Topics related to recommendations or questions on the design for HPCC Systems clusters

Interested in such CLI?

Poll ended at Sun Jun 03, 2018 9:21 pm

Yes
1
100%
No
0
No votes
 
Total votes : 1

Fri May 04, 2018 9:20 pm Change Time Zone

Hello!

Not sure where to post this...
I was wondering if anyone from the community feels like a CLI might be useful to ease creation, sharing and administration of HPCC Systems Clusters (be it Thor or Roxie or both)?
Think "vagrant", like vagrant init to initialize configuration file(s) for cluster to be brought up, and vagrant up to create actual cluster according to specs.

I am aware of the https://aws.hpccsystems.com/ and templates from Timothy Humphrey (tlhumphrey2?) on github (https://github.com/tlhumphrey2/EasyFastHPCCoAWS).
This CLI is inspired by both and I mostly use it to quickly boot up clusters (once one has a configuration file, it's fast to copy it over and tweak it) and run ECL code. Right now it only works with AWS.

One big benefit here is that I can commit those cluster configuration files for someone else to checkout and/or copy, and easily boot similar cluster and tweak as needed.
I personally use it extensively to run benchmarks and figure out best configuration within AWS (EBS volumes are everything but AWS-ome).

In a nutshell:
Code: Select all
mkdir mycluster
cd mycluster
# initializes the current directory to be an HPCC Systems environment by creating an initial cluster.config file if one does not already exist
hpcc-cluster init
# configure cluster
vi cluster.config
# (optional) validate configuration
hpcc-init validate
# create cluster
hpcc-cluster up
# start configmgr. Will open browser to http://....:8015/ page if started successfully
hpcc-cluster configmgr
# stop configmgr
hpcc-cluster configmgr
# first make sure hpcc-init stopped, then copy environment.xml file to all nodes
hpcc-cluster hpcc-init update
# now start cluster
hpcc-cluster hpcc-init start

# do stuff...

# stop cluster cause it's time to go to bed and I'd be wasting money just leaving it up and running...
hpcc-cluster halt

# after good long night's sleep, start cluster back:
hpcc-cluster resume


Another thing is does is use aliases for nodes to help target them.
For example, to ssh into the 6th slave (using consecutive private IP addresses):
Code: Select all
hpcc-cluster ssh @slave005


To simply ssh into the Master node:
Code: Select all
hpcc-cluster ssh


To open ECL Watch page:
Code: Select all
hpcc-cluster eclwatch



Here's the help page from this cli:
Code: Select all
  Usage: hpcc-cluster [options] [command]

  For manual, use man hpcc-cluster

  Options:

    -V, --version                 output the version number
    -d, --debug <level>           Specify log level (default: info)
    -p, --profile <profile>       Specify AWS Configuration Profile to use.
    -r, --region <region>         Specify AWS region to use (default: us-east-1)
    -h, --help                    output usage information

  Commands:

    init [options]                Initialize cluster configuration.
    up|create                     Create new cluster or Update existing cluster based on configuration.
    validate                      Validate template using cluster configuration. This is mostly for debugging purposes when updating the cluster template/configuration.
    resume                        Resume cluster previously halted.
    halt                          Halt current cluster. Cluster can be resumed thereafter.
    destroy|terminate             Destroy current cluster. Cluster CAN NOT be stopped nor resumed thereafter.
    status                        Display status of current cluster.
    help                          Display help.
    estimate                      Estimate the costs of your current configuration.
    eclwatch                      Open ECL Watch page.
    run [options] <target> <cmd>  Run command in target(s). Example: run slave* "sudo resize2fs /dev/xvdf".
    hpcc-init <cmd> [ip_or_node]  HPCC Cluster itself. Possible commands: start, stop, restart, status, stopAll (stops dafilesrv as well) and update which stops cluster, copy source/environment.xml file and push to all nodes.
    configmgr [ip_or_node]        Start/Stop HPCC Config Manager
    ssh [ip_or_node]              SSH into node of current cluster
    scp <source> <target>         SCP files from/to node. Just prefix remote with ":", like "scp local_file.txt @slave000:/tmp/remote_file.txt".


It doesn't show here but there's a lot of things happening behind the scene. For example, one can specify an ephemeral volume to be used on a node (say for mythor temporary/spill files in slaves) but those volumes need to be "re-setup" after stop/start of instance, and not when doing a soft reboot.
It's just not as easy as putting those in /etc/fstab, especially when one wants to use those in a RAID setting (which this tool supports).

Anyone interested in such CLI?
If there's enough interest I'll open source it.

Thanks!
Last edited by lpezet on Fri May 04, 2018 9:21 pm, edited 1 time in total.
lpezet
 
Posts: 53
Joined: Wed Sep 10, 2014 3:14 am

Fri May 04, 2018 9:21 pm Change Time Zone

Here's what a configuration file might look like:
Code: Select all
---
AWS:
  Profile: someprofile
  Region: us-east-1
  # run: "aws iam get-user" to get your username
  Username: first.last
  S3Bucket: bucket-where-cluster-files-will-be-uploaded
# If true, most operations won't trigger any AWS action, or will trigger AWS actions with "DryRun" parameter.
DryRun: true

# Used for notifications
Email: 'youremail@domain.com'

Vpc:
  # default or dedicated
  Tenancy: default
  # form of x.x.x.x/xx
  CidrBlock: 192.168.0.10/24
  # usually in the form of subnet-xxxxxxxx
  SubnetId: subnet-12345678
  # usually in the form of sg-xxxxxxxx
  SecurityGroupId: sg-12345678

Cluster:
  # 0-45
  Slaves: 1
  # usually 1 or 2
  Supports: 1
  # Name of cluster
  Name: hpccv3-fun-name-auto-generated
 
# Default settings applied to all type of instances (Master, Support, and Slave).
Instance:
  # Key Pair name from the region. Will be imported into instance and used when ssh into.
  KeyName: hpcc-cluster 
  # Depends on region. Best is to either check AWS documentation or try to create a simple instance with AWS Console and see the image id being used.
  # Some hints for now: us-east-1 (a) : ami-0b33d91d, us-east-2 : ami-c55673a0
  ImageId: ami-0b33d91d
  # Role to be used by EC2 instances. This is importat as resources will be downloaded using "aws" cli, which will use this role for permissions.
  IamRole: hpcc-cluster
  # Valid EC2 Instance Type for region AND availability zone (!!!). Check AWS documentation for list of valid types.
  Type: m5.xlarge
  Volumes:
    - # required
      DeviceName: /dev/xvdf
      # required if type != ephemeral. Size in GB
      Size: 1
      # optional, defaults to gp2. Valid EBS Volume Type: gp2, io1, st1, etc. Check AWS documentation for exhaustive and up-to-date list of valid values.
      # ephemeral is also possible for instance store.
      Type: gp2
      # optional, defaults to ext4. Values depend on OS but most supported ones are ext2, ext3, ext4, and xfs.
      FSType: ext4
      # optional, defaults to false. Either true or false
      Encrypted: true
      # required for io1 type volume.
      #Iops: 1000
      # optional. Where the device will be mounted.
      Mount: /volumes/vol-data
      # optional. Will create a symbolic link to the mount
      MapsTo: /var/lib/HPCCSystems/hpcc-data
      # optional. If part of an array, specify array device name
      #RaidDeviceName: /dev/md0
  # RAID devices
  # NB: RAIDs on ephemeral devices has not been tested yet.
  Raids:
    - # required
      DeviceName: /dev/md0
      # required
      Name: MyRaid0
      # required
      Level: 0
      # optional, defaults to ext4. Values depend on OS but most supported ones are ext2, ext3, ext4, and xfs.
      FSType: ext4
      # required. Where the device will be mounted.
      Mount: /volumes/vol-data
      # optional. Will create a symbolic link to the mount
      MapsTo: /var/lib/HPCCSystems/hpcc-data
     
MasterInstance:
  Volumes:
    - DeviceName: /dev/xvdf
      Size: 10
      Type: gp2
      Encrypted: true
      Mount: /volumes/vol-data
      MapsTo: /var/lib/HPCCSystems/hpcc-data
    - DeviceName: /dev/xvdz
      Type: gp2
      # Specify SnapshotId only or Size and Encrypted for new volume
      Size: 10
      Encrypted: true
      #SnapshotId: snap-12345678901234567
      Mount: /volumes/vol-dropzone
      MapsTo: /var/lib/HPCCSystems/mydropzone
     
SlaveInstance:
  # could be of a different type
  Type: m5.xlarge
  Volumes:
    - DeviceName: /dev/xvdf
      Size: 80
      Type: gp2
      Encrypted: true
      Mount: /volumes/vol-data
      MapsTo: /var/lib/HPCCSystems/hpcc-data
    - DeviceName: /dev/xvdg
      Size: 50
      Type: gp2
      Encrypted: true
      Mount: /volumes/vol-thor
      MapsTo: /var/lib/HPCCSystems/mythor
    - DeviceName: /dev/xvdh
      Size: 10
      Type: gp2
      Encrypted: true
      Mount: /volumes/vol-hpcc-mirror
      MapsTo: /var/lib/HPCCSystems/hpcc-mirror

SupportInstance:
  # could be of a different type
  Type: m5.xlarge
  Volumes:
    - DeviceName: /dev/xvdf
      Size: 80
      Type: gp2
      Encrypted: true
      Mount: /volumes/vol-data
      MapsTo: /var/lib/HPCCSystems/hpcc-data
lpezet
 
Posts: 53
Joined: Wed Sep 10, 2014 3:14 am

Mon May 07, 2018 12:13 pm Change Time Zone

I would be interested in your CLI. Do you have the code for it on github?
tlhumphrey2
 
Posts: 250
Joined: Mon May 07, 2012 6:23 pm

Mon May 07, 2018 7:23 pm Change Time Zone

This is very good.
Also we will provide generic tools to deploy HPCC cluster
1) Docker Compose.
2) Kubernetes.
2.1) local Virtualbox with minikube
2.2) AWS
2.3) Google Computing Engine if we have the environment
3) Juju charm
3.1) local Linux
3.2) AWS
3.3) Azure if we have the environment
3.4) Google Computing Engine if we have the environment

We have all of three in the past. But need some re-work and update.
ming
 
Posts: 20
Joined: Wed Dec 18, 2013 12:38 pm


Return to Clustering

Who is online

Users browsing this forum: No registered users and 0 guests

cron