For decades, the command line interface (CLI) has been a great way to run applications whether you are a developer going through unit testing or a user running an application. It provides a lower level of configuration and a higher level of transparency. Considering those two aspects of the command line tools, using CLI to deploy the HPCC Systems Cloud Native Platform can be a good starting point for new users, or an experienced one troubleshooting a deployment issue. This blog explores, from beginning to end, how to deploy the HPCC Systems Cloud Native Platform using the command line.  

Things you will need 

• A working computer that supports Linux, MacOS, or Windows OS. 
• An Azure account with sufficient credits. To obtain this, please go to www.azure.com or talk to your manager if you believe that your employer might already have a corporate one. 
• A code editor of your choice. Here, we will be using Visual Studio Code. 

In this tutorial, we will:

Install the necessary pre-requisites:

1. Install Azure CLI. 
2. Install Kubectl. 
3. Install Helm. 
4. Install VSCode. 
5. Install Git. 

Complete the Azure steps:

1. Create an Azure resource group. 
2. Add Azure network security rules. 
3. Create an Azure Kubernetes Service (AKS) cluster. 
4. Add an additional system node pool to the AKS cluster. 
5. Delete the existing default system node pool. 
6. Add a user node pool to the AKS cluster. 
7. Create an Azure storage account. 
8. Create a Kubernetes secret based on metadata from the storage account. 
9. Create an Azure private endpoint for the network. 
10. Create an Azure standard private load balancer. 
11. Create an Azure private link service. 

Integrate HPCC Systems:

1. Deploy the HPCC Systems storage chart. 
2. Deploy the HPCC Systems Platform chart. 
3. Deploy ELK (Elastic Search, Logstash and Kibana) HPCC Systems chart. 

Prerequisites:

Install Azure CLI 

Azure CLI is the command line tool from Azure that allows users to manage their subscriptions, create resources, etc. 

Please visit this web page for installation instructions: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli 

Install Kubectl 

Kubectl is the command line tool that allows users to manage their Kubernetes cluster. 

Please visit this web page for installation instructions: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli 

Install Helm 

Helm is a tool that allows you to manage Kubernetes applications. 

Please visit this web page for installation instructions: https://helm.sh/ 

Install VSCode  

VSCode or Virtual Studio Code is a code editor that allows users to edit documents. 

Please visit this web page for installation instructions: https://code.visualstudio.com/docs/setup/setup-overview 

Install Git 

Git is a version control tool that allows you to track historical changes of documents. 

Please visit this web page for installation instructions: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git 

Login to Azure from the Terminal 

1. Launch the terminal or command line 
2. Run: az login 
3. Follow the instructions

Create an Azure Resource Group 

An Azure resource group or RSG can be seen as a folder where a group of related resources live. For more on Azure resource groups please see https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview . 

az group create \ 
--name hpccdemo-rsg \ 
--location eastus2 \ 
--tags owner=’my_name’ owner_email=’my_name@lexisnexisrisk.com’ \ 
deployment-method=cli 

Create an Azure Kubernetes Service Cluster 

Azure Kubernetes Service is a service provided by Azure that offers Kubernetes as a service. It creates and manages the Kubernetes cluster for you. 

az aks create \ 
--name hpccdemo-aks \ 
--resource-group hpccdemo-rsg \ 
--node-vm-size Standard_A4_v2 \ 
--enable-managed-identity \ 
--tags owner=’my_name’ owner_email=’my_name@lexisnexisrisk.com’ \ 
deployment-method=cli 

Add an Additional System Node Pool 

The Azure Kubernetes Service automatically creates a default node pool called system node pool. By convention, you should schedule all Kubernetes system pods, such as kubelet, kube-proxy, and others, only on the system node pools, and application pods on user node pools only. Not doing so can cause odd behaviors.

az aks nodepool add \ 
--name sysnodepool \ 
--cluster-name hpccdemo-aks \ 
--resource-group hpccdemo-rsg \ 
--node-vm-size Standard_A4_v2 \ 
--mode System \ 
--enable-cluster-autoscaler \ 
--node-count = 2 \ 
--min-count = 1 \ 
--max-count = 2 \ 
--node-taints CriticalAddonsOnly=true:NoSchedule \ 
--tags owner=’my_name’ owner_email=’my_name@lexisnexisrisk.com’ \ 
deployment-method=cli 

Remove the Existing Default Node Pool 

The default node pool is not tainted to restrict application pods, which is the reason why we had to add a new system node pool above. Now, the default one needs to be deleted. 

az aks nodepool delete \ 
--name nodepool1 \ 
--cluster-name hpccdemo-aks \ 
--resource-group hpccdemo-rsg 

Add a User Node Pool to the AKS Cluster 

Currently, the AKS Cluster does not have a node pool on which to schedule the HPCC Systems pods. Before deploying the HPCC Systems Platform, at least one user node pool is needed. 

az aks nodepool add \ 
--name nodepool2 \ 
--cluster-name hpccdemo-aks \ 
--resource-group hpccdemo-rsg \ 
--enable-cluster-autoscaler \ 
--node-count = 4 \ 
--min-count = 2 \ 
--max-count = 4 \ 
--node-vm-size Standard_A4_v2 \ 
--mode User \ 
--tags owner=’my_name’ owner_email=’my_name@lexisnexisrisk.com’ \ 
deployment-method=cli 

Please Note: To learn more about Azure virtual machine pricing and types, please visit https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/ . 

Connect to the AKS from your local machine 

In order to deploy on the AKS cluster and manage it, a connection between the AKS and our local machine must be established. 

az aks get-credentials \ 
--name hpccdemo-aks \ 
--resource-group hpccdemo-rsg 

Please Note: Skip to Deploy the HPCC Systems storage chart section if you do not wish to use a storage account. Consequently, you will lose your data when the AKS is taken down. 

Working with the Storage Account Utility 

A storage account utility is provided in the helm-chart repository. We will be using that utility to create a storage account and a secret to link it to the Kubernetes cluster. 

The following code: 

• Clones the HPCC-Platform repo. 
•  Fetches all the available tags. 
• Checkouts a tag. 
• Creates a new directory outside of the HPCC-Platform. 
• Copies the sub-directories and files from HPCC-Platform/helm/examples/azure to the new directory that was created in step 4. 
• Copies HPCC-Platform/helm/hpcc/values.yaml to the directory that was created in step 4. 

Unix-based OS 

git clone https://github.com/hpcc-systems/HPCC-Platform.git\ 
git fetch --all 
git checkout community_8.4.22-1 \ 
mkdir ~/cloud \ 
cp -r HPCC-Platform/helm/examples/azure ~/cloud \ 
cp HPCC-Platform/helm/hpcc/values.yaml ~/cloud/azure 

Windows OS 

git clone https://github.com/hpcc-systems/HPCC-Platform.git  
git fetch --all 
git checkout community_8.4.22-1 
mkdir c:\cloud  
xcopy c:\HPCC-Platform\helm\examples\azure c:\cloud  
copy c:\HPCC-Platform\helm\hpcc\values.yaml c:\cloud\azure  

• Open the env-sa file that you copied in the new directory and set the arguments with the correct information. 

code ~/cloud/azure/sa/env-sa 

Example of the env-sa file 

# Azure subscription 
# SUBSCRIPTION= 
STORAGE_ACCOUNT_NAME=hpccdemosa 
SA_RESOURCE_GROUP=hpccdemo-sa-rg 
TAGS=’owner=”my_name” owner_email=”my_name@lexisnexisrisk.com”' 
# Set the same location as Kubernetes cluster 
SA_LOCATION=eastus 
SA_KEY_DIR=’./keys’ 
SA_SKU=Standard_LRS 

# These settings are also for generating PersistentVolume. 
# Settings in ../hpcc-azurefile/values.yaml will have higher priority. 
# If you will set these with "--set" for helm install you need set them 
# here also if you want to generate storage account, storage share and 
# Kubernetes secret. 
SECRET_NAME=hpccdemosecret 
SECRET_NAMESPACE="default" 
SHARE_NAMES="dalishare dllshare sashashare datashare lzshare" 

• Open the values.yaml file you copied in the new directory. Delete everything except the esp dictionary as shown below.

Code ~/cloud/azure/values.yaml 

By default, the service IPs are private. With the file open, set the visibility for eclwatch and eclqueries from local to global like in the below example. 

The new values.yaml file in the azure directory must look like the following:

esp: 
- name: eclwatch 
## Pre-configured esp applications include eclwatch, eclservices, and eclqueries 
application: eclwatch 
auth: none 
replicas: 1 
service:az 
## port can be used to change the local port used by the pod. If omitted, the default port (8880) is used 
port: 8888 
## servicePort controls the port that this service will be exposed on, either internally to the cluster, or externally 
servicePort: 8010 
## Specify visibility: local (or global) if you want the service available from outside the cluster. Typically, eclwatch and wsecl are published externally, while eclservices is designed for internal use. 
visibility: global 
## Annotations can be specified on a service - for example to specify provider-specific information such as service.beta.kubernetes.io/azure-load-balancer-internal-subnet 
#annotations: 
# service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "mysubnet" 
## You can also specify labels on a service 
#labels: 
# mylabel: "3" 
#resources: 
# cpu: "1" 
# memory: "2G" 
- name: eclservices 
application: eclservices 
auth: none 
replicas: 1 
service: 
servicePort: 8010 
visibility: cluster 
#resources: 
# cpu: "250m" 
# memory: "1G" 
- name: eclqueries 
application: eclqueries 
auth: none 
replicas: 1 
service: 
visibility: global 
servicePort: 8002 
#resources: 
# cpu: "250m" 
# memory: "1G" 
- name: esdl-sandbox 
application: esdl-sandbox 
auth: none 
replicas: 1 
service: 
visibility: local 
servicePort: 8899 
#resources: 
# cpu: "250m" 
# memory: "1G" 
- name: sql2ecl 
application: sql2ecl 
auth: none 
replicas: 1 
service: 
visibility: local 
servicePort: 8510 
#domain: hpccsql.com 
#resources: 
# cpu: "250m" 
# memory: "1G" 

Open the values.yaml in the hpcc-azurefile directory

code ~/cloud/azure/hpcc-azurefile/values.yaml 

1. Uncomment the following key-value pairs: secretName, secretNamespace, sku and shareName. 
2. Set secretName to hpccdemosecret. Example: secretName=hpccdemosecret  
3. Save the file 

Example of the values.yaml in the hpcc-azurefile directory

# Default values for hpcc-azurelfile. 
common: 
mountPrefix: "/var/lib/HPCCSystems" 
secretName: "hpccdemosecret" 
secretNamespace: "default" 
 
 
planes: 
- name: dali 
subPath: dalistorage 
size: 1Gi 
category: dali 
sku: "Standard_LRS" 
shareName: dalishare 
- name: dll 
subPath: queries # cannot currently be changed 
size: 1Gi 
category: dll 
rwmany: true 
sku: "Standard_LRS" 
shareName: dllsshare 
- name: sasha 
subPath: sasha 
size: 1Gi 
rwmany: true 
category: sasha 
sku: "Standard_LRS" 
shareName: sashashare 
- name: data 
subPath: hpcc-data # cannot currently be changed 
size: 3Gi 
category: data # NB: all "data" planes will be auto mounted by engine components and others that require access to data 
rwmany: true 
sku: "Standard_LRS" 
shareName: datashare 
- name: mydropzone 
subPath: dropzone 
size: 1Gi 
rwmany: true 
category: lz 
sku: "Standard_LRS" 
shareName: lzshare 

HPCC Systems

Create the Storage Account and secret

~/cloud/azure/sa/create-sa.sh  
~/cloud/azure/sa/create-secret.sh 

Deploy the storage helm chart

helm install azstorage ~/cloud/azure/hpcc-azurefile 

Deploy the HPCC Systems Platform helm chart

The above command will add the HPCC helm repository to your local helm repository, then deploys the platform chart. 

helm repo add hpcc  https://hpcc-systems.github.io/helm-chart \ 
helm install myhpcck8s hpcc/hpcc –set global.image.version=’9.6.20-rc1’ -f ~/cloud/azure/values-retained-hpcc-azurefile.yaml -f ~/cloud/azure/values.yaml 

Access ECLWatch

1. List running services 

Kubectl get svc 

2. Copy ECLWatch external IP and paste it in your browser followed by :8010. For example: 20.90.16.76:8010 

References 

For more on Helm commands:  

https://helm.sh/docs/helm

For more on Kubectl commands:  

https://kubernetes.io/docs/reference/kubectl/cheatsheet

For more on Azure commands:  

https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest

Meet the author