Home > Resources > Getting Started with the HPCC Systems Cloud Native Platform Using Terraform and Azure

Getting Started with the HPCC Systems Cloud Native Platform Using Terraform and Azure

Image showing the Terraform Logo

This blog provides step-by-step instructions, showing how to deploy the HPCC Systems Cloud Native Platform into Azure using Terraform open source and additional modules from the LexisNexis Risk Solutions Terraform open-source registry. No previous knowledge of Terraform, Kubernetes or Azure is required.

Please Note…

The modules used in this blog were designed to be used as a template for development environments. There is no guarantee that these modules will meet your organization’s cloud policies. It is advised that you speak with your site reliability engineering team before forking the terraform-azurerm-hpcc repository and modifying it for use in a production environment.

**********************

Manually creating your cloud infrastructure, storage needs, installing the HPCC Systems helm chart and other features by hand is a particularly good exercise for beginners. However, as you start deploying every day, it may become tedious and error prone. You may find:

  • Inconsistencies in your configurations as your deployments become exponentially more difficult
  • It becomes time consuming as your need for additional resources grows
  • Complying with security requirements in a corporate setting becomes a potential minefield

It most likely would not take long to ask yourself whether there is a better way to do it. The good news is that there are multiple IaC (Infrastructure as Code) tools available which make the entire deployment process feel like waving a magic wand. One of these tools is Terraform.

Before you start…

A little preparation is required to make sure you have everything available to complete this blog tutorial. You will need:

  • A working computer that supports Linux, MacOS, or Windows OS
  • An Azure account with sufficient credits. Go to www.azure.com, or talk to your manager  if you believe your organisation already have one that is available to use
  • A code editor of your choice. This blog uses VS Code
  • A web browser to access the ECL Watch user interface

Installing the Third Party Tools

A number of third party tools are needed to complete this blog tutorial and since third party tools may be regularly updated, this blog does not include specific installation instructions which may change over time. However, you can get the most up to date instructions from the various providers using the links supplied.

Installing Git

Git is widely used so you may already have this installed on your machine. However, if you are unfamiliar with Git, it may be helpful to know that it is a version control tool that allows the tracking of historical changes made to files. The instructions in this blog require the use of Git to clone (download) the terraform-azurerm-hpcc-aks module from GitHub. So if you do not already have Git installed, use this link to get started installing Git.

Installing VS Code (Optional)

Visual Studio Code is a code editor used in this tutorial to edit the configuration files. If you do not already have VS Code installed, use this link for setup instructions, or you may prefer to continue to use your editor of choice.

Installing Kubectl

Kubectl is the Kubernetes command-line tool that allows you to manage a Kubernetes cluster from your local machine. In this tutorial, it is used to inspect and manage the resources, for example, the pods, services, persistent volumes etc, and also to view the logs. If you do not already have Kubectl installed, use this link for installation instructions.

More information on Kubectl command is available on this cheat sheet.

Installing Azure CLI

Azure CLI is the command-line tool from Azure that allows you to login to your Azure cloud account and enable your subscription to be used by the Terraform Azure resource manager provider. If you do not already have this installed, use this link for more information.

More information on Azure CLI commands is available on this cheat sheet.

Installing Terraform

Terraform is an Infrastructure as Code (IAC) tool. The entire infrastructure, such as the virtual network, Kubernetes, storage and others will be codified in the Terraform language.

Version 0.15.0 or later is required. If you do not already have it installed, use this link to access the installation tutorial.

Installing Python3

Python is used to query data between the modules automatically, removing any need for you to follow a manual process. Use this link to install Python3.

Installing Google Chrome

Google Chrome is used as the preferred web browser in this tutorial. Other web browsers may be used as alternatives, but if you want to follow along using Google Chrome and you do not already have it installed, use this link to get set up.

Setting up your Azure Subscription

First you need to login to Azure from the terminal using the following command:

 az login

Then follow the instructions as provided. Once logged in, check that the currently active Azure subscription is the one you want to use (it may not have been enabled by default). To do this, use the following command:

 az account list | grep -A 3 ‘”isDefault”: true’

If the account listed is not the one you want to use, use this command to change to the required account:

 az account set --subscription <my-subscription-name>

Working with the Terraform Code

Once all the third party tools are installed and your Azure account is correctly configured, the next step is to clone the Terraform module from GitHub.

Cloning the Terraform Repository

The terraform-azurerm-hpcc repository contains three modules which must be deployed in the following order:

  1. terraform-azures-hpcc/modules/virtual_network
    The Network module must be deployed first

  2. terraform-azurerm-hpcc-storage
    The Storage module creates a storage account using blob storage, which stores your HPCC Systems platform data. The stored data are durable, highly available, secure and scalable

  3. terraform-azurerm-hpcc-aks
    The AKS module creates the Kubernetes services and deploys the HPCC Systems Platform

To clone the repository:

  1. Open your command line or terminal
  2. Determine where you want to store this project on your machine by changing directory to the chosen location.
    Note: Choose a location that is easy to remember, such as Desktop with a nested folder, for example, Desktop/cloud/azure/terraform/
  3. Run the following command:
 git clone https://github.com/hpcc-systems/terraform-azurerm-hpcc.git

The following is a list of modules to deploy and their locations:

Module Location
Virtual Network terraform-azurerm-hpcc/modules/virtual_network
Storage Account terraform-azurerm-hpcc/modules/storage_account
AKS terraform-azurerm-hpcc

​​​​​​Note: The Network module must be deployed first, followed by the Storage module. Finally, the AKS module, which is at the root of the repository, can be deployed. These modules automatically call other dependent modules upon initialization.

Configuring the admin.tfvars files for each module

After cloning the repository, the next step is to configure each module by editing the admin.tfvars files.

To do this, move into each module, take a copy of the examples/admin.tfvars file and configure it. For example, to configure and deploy the network module:

  1. Change directory to terraform-azurerm-hpcc/modules/virtual_network
  2. Copy admin.tfvars from ./examples to ./virtual_network:

Unix Based Operating System:

 cp examples/admin.tfvars ./admin.tfvars

Windows Based Operating System:

 copy examplesadmin.tfvars .admin.tfvars

On your command line (assuming your editor of choice is configured to be launched from the command line or terminal), edit the copied admin.tfvars. For example, to edit with VS Code, use:

 code terraform-azurerm-hpcc/modules/virtual_network/admin.tfvars

Review each object block or argument in the file and set it using your preferred values. Once you are done, save the admin.tfvars files.

Now repeat this process to edit the admin.tfvars located in storage account and AKS modules, but note that the AKS module settings are within the root repo directory (terraform-azurerm-hpcc) not under modules.

Note: Please read the README.md in the HPCC Systems GitHub repository to understand the arguments in the admin.tfvars file and others that are also supported. There is also an optional argument list at the bottom of the admin.tfvars files. After each new git pull, it is good practice to look at the example/admin.tfvars file for new changes and update your custom admin.tfvars file accordingly. The admin blocks contain information specific to the user deploying the cluster. So you must change that block to include your specific information.

An example of admin.tfvars for the Virtual Network Module

admin = {
  name  = "hpccdemo"
  email = "hpccdemo@example.com"
}

metadata = {
  project             = "hpccdemo"
  product_name        = "vnet"
  business_unit       = "commercial"
  environment         = "sandbox"
  market              = "us"
  product_group       = "contoso"
  resource_group_type = "app"
  sre_team            = "hpccplatform"
  subscription_type   = "dev"
}

tags = { "justification" = "testing" }

resource_group = {
  unique_name = true
  location    = "eastus2"
}

Virtual Network Module Argument Definitions

The following are available and more information is available in the terraform-azurerm-hpcc readme.me file.

admin Block – (Required)

The admin block contains information on the user deploying the cluster. This is used as tags and part of some resource names to identify who deployed a given resource and how to contact that user. This block is required. The following arguments are supported by the admin blocks:

  • name (Required) – The name of the user who is deploying the cluster
  • email (Required) – The email of the user who is deploying the cluster

metadata Block – (optional)

The arguments in this block are used as tags and part of the resource names. This block can be omitted when disable_naming_convention is set to true.

  • project (Required) – The name of your project
  • product_name (Optional) – The name of the product
  • business_unit (Optional) – The name of the business unit
  • environment (Optional) – The name of the environment
  • market (Optional) – The name of the market
  • product_group (Optional) – The name of the product group
  • resource_group_type (Optional) – The name of the resource group
  • sre_team (Optional) – The name of the Site Reliability Engineer (SRE) team
  • subscription_type (Optional) – The type of subscription being used

tag attribute – (Optional)

Used for additional tags which must be in key value pairs.

resource_group Block – (Required)

The resource_group block creates a resource group (like a folder) for your resources and supports the following arguments:

  • unique_name (Optional) – Render a unique name for the resource group. Possible values are true/false
  • location (Required) – Set the region in which to deploy the cluster

An example of admin.tfvars for the storage account

admin = {
  name  = "hpccdemo"
  email = "hpccdemo@example.com"
}

metadata = {
  project             = "hpccdemo"
  product_name        = "vnet"
  business_unit       = "commercial"
  environment         = "sandbox"
  market              = "us"
  product_group       = "contoso"
  resource_group_type = "app"
  sre_team            = "hpccplatform"
  subscription_type   = "dev"
}

tags = { "justification" = "testing" }

resource_group = {
  unique_name = true
  location    = "eastus2"
}

storage = {
  access_tier = "Hot"
  account_kind = "StorageV2"
  account_tier = "Standard"
  replication_type = "LRS"
  enable_large_file_share = true
  # auto_approval_subscription_ids = []

  quotas = {
    dali = 1
    data = 3
    dll = 1
    lz = 1
    sasha = 3
  }
}

Storage Account Module Argument Definitions

storage Block – (Required)

The storage block deploys the storage chart for HPCC Systems. This block is required, but some of its arguments are optional. The following arguments are supported:

  • access_tier (Optional) – Defines the access tier to use for the kind of account. Valid options are Hot and Cold. Defaults to Hot
  • account_kind (Optional) – Defines the kind of storage to use. The current valid option for HPCC Systems is StorageV2. Defaults to StorageV2
  • account_tier (Required) – Defines the tier to use for the storage account. Valid Options are Standard and Premium
  • replication_type (Required) – The type of replication to use for the storage account. Valid options are LRS, GRS, RAGRS, ZRS, GZRS, RAGZRS
  • enable_large_file_share (Required) – To enable large file share
  • auto_approval_subscription_ids (Optional) – Subscription IDs to grant access if different that current subscription
  • quotas – (Required) – The quotas in gigabytes for the storage shares as defined below

quotas Block

The quotas block supports the following arguments:

  • dali (Required) – Dali storage share
  • data (Required) – DATA storage share
  • dll (Required) – DLL storage share
  • lz (Required) – Landing-zone storage share
  • sasha (Required) – Sasha storage share

An example of admin.tfvars for the AKS Module

admin = {
  name  = "hpccdemo"
  email = "hpccdemo@example.com"
}

metadata = {
  project             = "hpccdemo"
  product_name        = "contosoweb"
  business_unit       = "commercial"
  environment         = "dev"
  market              = "us"
  product_group       = "contoso"
  resource_group_type = "app"
  sre_team            = "hpccplatform"
  subscription_type   = "dev"
}

resource_group = {
  unique_name = true
  location    = "eastus2"
}

node_pools = {
  system = {
    vm_size                      = "Standard_B2s"
    node_count                   = 1
    enable_auto_scaling          = true
    only_critical_addons_enabled = true
    min_count                    = 1
    max_count                    = 2
    subnet                       = "private"
  }

  addpool1 = {
    vm_size             = "Standard_B2ms"
    enable_auto_scaling = true
    min_count           = 1
    max_count           = 2
    subnet              = "private"
  }

  addpool2 = {
    vm_size             = "Standard_B2ms"
    enable_auto_scaling = true
    min_count           = 1
    max_count           = 3
    subnet              = "public"
  }
}

hpcc = {
  version   = "8.2.12-rc1"
  name      = "myhpcck8s"
  atomic    = true
  timeout   = 340
  # namespace = ""
}

storage = {
  default = false
  # chart = ""
  # values = []
  /*
  storage_account = {
  location = "eastus"
  name = "demohpccsa3"
  resource_group_name  = "app-storageaccount-sandbox-eastus-48936"
  # subscription_id = ""
  }
  */
}

elk = {
  enable = true
  name   = "myhpccelk"
}

AKS Module Argument Definitions

The following are available and more information is available in the terraform-azurerm-hpcc-aks readme.me file.

node_pools Block – (Required)

The node_pools block defines the system or default node pool and additional user node pools. This block is required and defined as follows:

  • system (Required) – The system or default node pool. This node pool hosts the system pods by default. The possible arguments for this block are defined below.
  • addpool (Required) – The additional node pool configuration. This block name is changeable and must be unique across all additional node pools. At least one additional node pool is required. The possible arguments for this block are defined below.

The system and addpool blocks – (Required)

The system and addpool blocks are defined by the following arguments:

  • vm_size (Optional)  -The size of the Virtual Machine, such as Standard_A4_v2
  • node_count (Optional) – The initial number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000 and between min_count and max_count
  • enable_auto_scalling (Optional) – Determines whether the Kubernetes Auto Scaler should be enabled for this Node Pool. Defaults to false
  • only_critical_addons_enabled (Optional) – Schedule system pods only on the system node pool
  • min_count (Optional) – The minimum number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000. Can only be set when enable_auto_scalling is set to true
  • max_count (Optional) – The maximum number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000. Can only be set when enable_auto_scalling is set to true
  • subnet (Required) – The subnet for the node pool

hpcc Block – (Required)

The hpcc block deploys the HPCC Systems helm chart. The hpcc block supports the following arguments:

  • name (Required) – The name of the cluster
  • namespace (Optional) – The namespace of the cluster. Defaults to the current Terraform workspace name
  • version (Optional) – The version of the chart. Either version or chart must be set
  • values (Optional) – Additional YAML configuration files for the deployment.

See the terraform-azurerm-hpcc-aks readme.me file for more optional attributes.

storage Block – (Optional)

The storage block supports the following arguments:

  • default (Required) – When set to true, the default storage provided by the Azure Kubernetes Service is used in place of a storage account
  • chart (Optional)  – A custom HPCC Systems Storage chart to deploy
  • values (Optional) – Additional YAML configuration files for the HPCC Systems Storage deployment
  • storage_account (Optional) Storage account metadata. The accepted arguments for this block are defined below

storage_account Block – (Optional)

This block should only be set if you already have a storage account outside of this Terraform deployment that you want to link to the AKS, instead of deploying a new one. In this blog, we will refer to a foreign storage account as a storage account that was not created by the storage account module of the repository, or where the outputs.json file in the bin directory has been lost. Information about your resources can be found on the Azure portal.

The storage_account block is defined by the following arguments:

  • name (Required) – The existing storage account name
  • resource_group_name (Required) – The existing storage account resource group name
  • location (Required) – The region where the foreign storage account is located
  • subscription_id (Optional) – The existing storage account subscription id. Used only when the subscription of the storage account is different to the subscription of the cluster to be linked to it.

elk Block – (Optional)

The elk block deploys the ELK (Elastic Logstash and Kibana) chart for HPCC Systems. The following arguments are supported:

  • enable (Optional) – Determines whether the ELK chart should be deployed? Valid values are true and false.

Other Optional Arguments for the admin.tfvars

The following arguments are independent of any other block:

  • expose_services (Optional) – Expose ECLWatch to the internet. Valid values are true and false
  • auto_connect (Optional) – Automatically connect to the Kubernetes cluster from the host machine. Valid values are true and false
  • image_root (Optional) – Root of the image other than hpccsystems
  • image_name (Optional) – Name of the image other than platform-core
  • image_version (Optional) – Version of the image. Must only be set when using an HPCC Systems Platform custom chart (Not storage or ELK custom chart)
  • tags (Optional) – Additional tags
  • disable_helm (Optional) – Disable Helm deployments by Terraform. This is reserved for experimentation with other deployment tools like Flux2. Valid values are true and false
  • disable_naming_conventions (Optional) – Disable naming conventions. Valid values are true and false
  • auto_launch_eclwatch (Optional) – Automatically launch the ECLWatch user interface upon the success of the deployment

Note: For additional supported arguments for the blocks listed above, please see the terraform-azurerm-hpcc-aks readme.me file.

Initializing the Terraform Modules

After configuring the environment, the next step is to initialize the root module. Initializing a Terraform module is similar to initializing a Git repo. You only have to do it once. The only time it is necessary to do it more than once is when the source of one of the modules is changed, although, running it multiple times is safe. The Terraform init command declares the current working directory as the root or calling module for the entire module. During this operation, Terraform downloads all the child modules from their sources. More information on terraform init is available in the Terraform CLI Documentation.

To initialize the modules, move to each module directory and run the following command:

 terraform init

Generating a Terraform Plan and Running Terraform Apply

The next step is to generate a plan for your configuration. Since a Terraform Apply will also validate the Terraform code and generate an infrastructure plan, generating a separate plan is not necessary. Simply, run the terraform apply command in each of the three module directories:

 terraform apply –var-file=admin.tfvars

Then scan through the generated plan before accepting or denying it. A Terraform plan gives a summary of the resources that are going to be added, changed or deleted.

Note: The Terraform apply process will not end until all declared resources are ready to be used and the process may take some time for Azure to create those resources.

Connecting to your Cluster from your Local Machine

To connect to the cluster, use the Azure CLI. An Azure command is printed out at the end of the Terraform apply command upon success. You can skip this step if auto-connect and auto-launch-eclwatch are set to true in admin.tfvars.

  1. Copy the az aks get-credentials command that was printed out
  2. Paste the command in your terminal or command-line and press Enter
  3. Accept to overwrite your current context if you are prompted to do so

To verify that the connection was successful, list the pods using the following command:

 kubectl get pods

Getting to the ECLWatch Page

To access the ECL Watch page, use the following command in your terminal:

 kubectl get svc

Find the ECL Watch service name and copy the external-IP. Paste the copied IP into to your browser followed by the port (for example, 240.21.34.10:8010).

Note: This same external-IP is used to connect to ECL Watch from the IDE or VS Code plugin.

Destroying the AKS Cluster

It is important to keep in mind that the AKS module must be destroyed before the virtual network module.

Attempting to destroy the virtual network before the AKS, will start destroying your output.json file, which contains some of the deployment metadata that is needed by the storage account and AKS modules and this will be followed by the destructions of the subnets. Since AKS will still be deployed on the subnets, the destruction will fail leaving your Net deployment in an odd state.

There is no cost associated with leaving your virtual network running. As for the storage account, you should consider destroying it if the stored data is dispensable.

However, always destroy your AKS when you do not plan to run any jobs to reduce costs. To bring it back up, do a terraform apply as shown earlier in this blog.

Like the Terraform apply command, the destroy command also auto-generates a plan. Change directory to terraform-azurerm-hpcc, using the following command:

 cd terraform-azurerm-hpcc

Then run the destroy command:

 terraform destroy –var-file=admin.tfvars

Once this has completed, scan through the plan and accept it following the prompt.

This process can also be used to destroy the storage account and virtual network. The virtual network can stay up, since Azure only charges for data that passes through the Net in some instances.

Deploying a Custom Image vs an Official One

When deploying a custom image, the image version must be set against the image_version argument and the image root must be set against the image_root argument. Other arguments, such as image_name may also be needed based on whether the image name has been changed from platform-core to something else.

In comparison, image arguments are not needed for official image deployments. By default, each chart version contains the corresponding image version. For example, setting the version argument in the hpcc block to 8.4.10 would automatically pull image version 8.4.10. In other words, all you need to do to deploy an official image is set the version argument in the hpcc block to the image version you wish to deploy.

Deploying a Custom HPCC Systems Chart vs an Official One

When it comes to using a custom chart for a deployment, the Terraform module intuitively looks at whether a value is set against the chart argument in the hpcc block. The chart argument expects an absolute path to the chart directory or a relative path to the terraform-azurerm-hpcc directory, for example, /home/user/hpcc-helm/helm/hpcc. Other arguments, such as image-version, might also be needed to indicate the image version corresponding to the custom chart.

Deploying Multiple Clusters

Workspaces allow you to deploy multiple clusters with each of them having their own states. A great use case for workspaces is to use one workspace per feature function.

For example, when testing a new feature with JIRA number HPCC-0241, you would create a new workspace named HPCC-0241 and deploy from that workspace. The workspace name would then be used as part of your Azure resource names and tags.

To do this (based on this example) using your command line tool, change directory to terraform-azurerm-hpcc using the following command:

 cd terraform-azurerm-hpcc

Create a new Terraform workspace:

 terraform workspace new HPCC-0241

You can now proceed with the normal deployment steps.

Cheat Sheet

The following is a quick guide to some of the more commonly used commands:

Auto approve Terraform apply command:

 terraform apply –var-file=admin.tfvars -auto-approve

Auto approve the destroy command:

 terraform destroy –var-file=admin.tfvars -auto-approve

Automatically connect to the Kubernetes cluster (admin.tfvars argument):

 auto_connect = true

Disable all helm deployments (admin.tfvars argument):

 disable_helm = true

Use disable_helm to destroy hpcc, storage and elk and leave Kubernetes cluster — For troubleshooting (admin.tfvars argument):

 disable_helm = true

List Terraform workspaces:

 terraform workspace list

Show the name of the currently selected workspace:

 terraform workspace show

Delete Terraform workspace:

 terraform workspace delete <my-undesired-workspace-name>

Switch to a Terraform workspace:

 terraform workspace select <my-new-workspace-name>

Show Advisor recommendations:

 terraform show

Frequently Asked Questions

Can I link an existing storage account with stored data?

You can always link your own storage account to the AKS without deploying the storage account module as long as you have the name of the storage, its resource group name and subscription ID if this is different from the AKS subscription.

How do I deploy on a foreign VNet instead of deploying the virtual network module?

To deploy on a foreign VNet or existing VNet, use the virtual_network block found in the admin.tfvars of the storage_account and AKS modules:

virtual network = {
   private_subnet_id = ""
   public_subnet_id = ""
   route_table_id = ""
   location.    = ""
}

How does vm_size differ from the values set in the quotas block?

The vm_size set the size of the nodes in the node pool while the quotas block set the size of the file shares, which hold the HPCC Systems Platform data.

How do I configure HPCC Systems components and services, such as Thor, Roxie, Dali, DFU, ECLCCServer, Certificates, etc.?

Create your own values.yaml file using the following instructions:

  1. Navigate to https://github.com/hpcc-systems/HPCC-Platform/blob/8.6.0/helm/hpcc/values.yaml
  2. Select the tag that corresponds to the version set in the hpcc block in the admin.tfvars file, or simply by changing the version number in the URL to the corresponding version
  3. Copy the entire content of the values.yaml or the portion of the configuration required
  4. Paste the copied values.yaml content into your new custom yaml file
  5. Save the custom file

How do I access HPCC Systems services, such as ECLQueries?

In your command line, run:

 kubectl get svc

Then copy the external IP of the ECLQueries followed by the port number in your browser, for example: 20.0.12.3:8002.

More Information

Learn more about the commands used in this blog and more:

*********************************************************

Image showing Godson Fortil

Godji Fortil is a Software Engineer I in HPCC Systems Development, at LexisNexis Risk Solutions. He joined the HPCC Systems Platform team in 2018 after completing a bachelor’s degree in Computer Science at Florida Atlantic University. Godji is involved in many different projects including testing, user support, site reliability and more recently, supporting our cloud native platform development. He has also mentored students joining the HPCC Systems Intern Program and presented at the 2021 HPCC Systems Community Day Summit about A Simple HPCC Systems Cloud Deployment for Open-Source Users, alongside our LexisNexis Risk Solutions Group colleague, Xiaoming “Ming” Wang.

Tags
Community Content
Cloud
Features & Tools
Getting Started