Running HPCC Systems Platform on Microsoft Hyper-V
If you are a PC user and want to run the HPCC Systems Platform, the simplest and most natural environment may be to use a Hyper-V Virtual Machine. Hyper-V is standard on many versions of Windows, and tends to work better than most add-on virtualization systems. It also has good support for multiple CPUs, and is quite reliable and performant.
It is quite easy to install and run the HPCC Systems Platform on Hyper-V, and is in many ways preferable to using the HPCC-provided VM image. A multi-node HPCC-Environment — with multiple Thor slave processes on a single computer — can be set up in a few minutes, assuming your personal PC supports Hyper-V.
We use Hyper-V to create a Virtual Machine (VM) with the well-supported and mature Ubuntu 20.04 operating system. Then we go through the standard HPCC System Platform installation process.
Enable Hyper-V on Windows
Hyper-V enables running multiple operating systems as virtual machines on Windows. Click HERE for more details about Hyper-V. If you’re already using Hyper-V on your system, you can skip this step.
First of all, let’s make sure Hyper-V is enabled on your Windows machine by following below steps:
- Right click on the Windows button and select ‘Apps and Features’.
- Select ‘Programs and Features’ on the right under ‘Related Settings’.
- Select ‘Turn Windows Features on or off’.
- Choose Hyper-V and click OK.
Create Linux Virtual Environment
Now let’s create Ubuntu virtual environment in Hyper-V by following below steps. Here we use Ubuntu 20.04.
1. Open Hyper-V Manager by typing ‘Hyper-V Manager’ in the search window and click on the best match.
2.In Hyper-V manager, click on the ‘Action’ button and choose ‘Quick Create’ option from the drop-down menu.
3. From the ‘Create virtual machine’ window, choose ‘Ubuntu 20.04’ and then click on the ‘Create Virtual Machine’ button at the bottom.
4. Follow the guide to finish installing Ubuntu. Click HERE for more details about the guide to install Ubuntu. Once created, it will be listed in the Virtual Machines list. Here we created a virtual machine named ‘Ubuntu 20.04’.
To fully utilize the resources such as logical processors in your local machine, we can configure multi-processor support for your VM in Hyper-V manager. Click HERE for more details.
5. Double click on the newly created Ubuntu 20.04 virtual environment and click on the Start button to connect the virtual machine.
6. Once connected, you will see below window.
Download and Install HPCC Systems
1. Click on the FireFox on the left panel to open FireFox browser. Type in the URL https://hpccsystems.com/download in the search window to direct to the HPCC Systems Download page. In the Download page, choose ‘Ubuntu 20.04 LTS’ as OS and click on the Download button down below to download the installer.
2. Once finished downloading, double click on the installer, which will open the Ubuntu software installation window.
3. In the Ubuntu software installation window, click ‘Install’ button to install HPCC Systems in your local environment. Once successfully installed, you will see the ‘Install’ button turns to ‘Remove’ button.
To fully utilize the processors assigned to the virtual machine, we can setup multiple slaves per node in HPCC Systems. The rule-of-thumb is to setup the number of slaves per node as the same number of logical processors in the host machine. Click HERE for more details of how to setup multi-slaves HPCC Systems.
Working with HPCC Systems Locally
1. Once installation is finished, type below command in the terminal to start the system.
sudo systemctl start hpccsystems-platform.target
2. Open FireFox browser and type ‘localhost:8010’ in the search window. If HPCC Systems is installed correctly, you will see ECL Watch home page in the browser as shown below. ECL Watch is a web-browser based interface for users to monitor ECL jobs and cluster servers in one spot. Click HERE for more detail about ECL Watch.
3. Now HPCC Systems is installed in your local environment, you can quickly try out some ECL code in ECL Watch Playground by clicking on the ‘ECL’ button (the grey gear shape button at the top) and click on the ‘Playground’ tab. Click HERE for more detail about ECL Playground.
Clear the code in the Edit window and type in your own ECL code to try out the magic of HPCC Systems. Or you can install ECL IDE or VS Code ECL extension like other ECL programmers to save and edit ECL code locally without losing the source code! Click HERE to know more about ECL IDE and HERE about VS Code ECL extension.
4. Lastly, if you want to shut down HPCC Systems in the VM, simply type below command in the terminal.
sudo /etc/init.d/hpcc-init stop
Share Files between Hyper-V Virtual Machine and Windows
You may have found that you are unable to copy and paste files between Virtual Machine and Windows directly. In this section, we will introduce how to setup share folders to transfer files between the virtual machine client in Hyper-V and Windows host.
First of all, let’s create a new local user on PC by following below steps:
1. Search ‘Manage your account’ in the search window and choose the best match to open Accounts window.
2. In the Accounts window, click ‘Other users’ on the left side and click on ‘Add someone else to this PC’ under Other users section on the right. Follow the guides to create a new local user account. Here we created a new local account with username ‘Ubuntu20’. Click HERE for more details about how to create a local user account in Windows.
3. Create a local share folder in PC. Here we created a share folder named VMshare. The file path is C:\VMShare.
4. Right click on the new local share folder we just created and choose Properties. In the Properties window, click on the Sharing tab and then click on Share button to open the Network access window.
5. In the Network access window, search the new local username and click Add to add to the share list. Here, we search the newly created user ‘Ubuntu20’ and added to the share list. Make sure change the Permission level to Read/Write. Then click Share button at the bottom.
6. Create a ‘Share’ folder under ‘Home’ directory.
7. Open Terminal and install cifs-utils by typing below command.
8. Type below command to create a shell script named mountShare.sh.
echo ‘sudo mount -t cifs - o username=USERNAME //HOSTNAME/VMShare ~/share’ > mountShare.sh
Change USERNAME to the new local username created previously. ChangeHOSTNAME to the hostname of your local machine.
9. In the terminal, type command ‘chmod +x mountShare.sh’ to make the shell script executable:
10. Type command ‘./mountShare.sh’ to run the shell script:
It will ask for the password for the new local user created previously. Type in the password and press Enter.
Now in the Home folder, you will see the ‘Share’ folder has a lock on it. It means you can transfer files back and forth between the client and the host freely.Files in the ‘Share’ folder in VM will sync with the files in the VMshare folder in Windows.
Utilize multiple processor for your Virtual Machine in Hyper-V
Hyper-V multi-processor support lets you control the number processor in a virtual machine. This allows a overall performance gain for the operations in the virtual environment.
To configure multi-processor in Hyper-V, open Hyper-V manager and right click on the virtual machine. In the pop-up menu, click on Settings.
In Settings, click on Processor. On the right side, adjust ‘Number of virtual processor’ needed for the virtual machine. Then click OK to confirm the setting. Note that the number of virtual processors cannot exceed the number of available physical processors in the host machine. You can see how many resources available from ‘Percent of total system resources’ in the Resource control section.
Configure multi-slaves in HPCC Systems
In the above section, we introduced how to setup multi-processor for your virtual machine. Here we will introduce how to configure HPCC Systems to fully utilize the assigned resource via multi-slaves.
In HPCC Systems, the number of slave per node can be configured in the system configuration file environment.xml under the path /etc/HPCCSystems/.
First, let’s open the terminal in the virtual machine. In the terminal, type below command to make sure the configuration file allows write access.
Then, type below command to edit the configuration file environment.xml.
In the editor, search ‘SlavesPerNode’ to change the number of slaves per node. The default value is 1. As mentioned previously, a rule-of-thumb is to setup the number of slaves as the same number of logical processors of your host machine. Since my host machine has 4 logical processor, let’s change it to 4 and save the file. Adjust the number based on your host machine.
Now your local HPCC Systems can be more efficient than before via multi-slave configurations. Of course, the requirement is you have to configure multi-processor support in Hyper-V first. Click HERE for details about how to setup multi-processor support in Hyper-V.