Home > Resources > Features new and improved in HPCC Systems 8.2.0 Gold

Features new and improved in HPCC Systems 8.2.0 Gold

Earlier in 2021, we announced the arrival of HPCC Systems 8.0.0 Gold, which heralded the release of our new Cloud Native Platform, which was made available for Cloud performance evaluation by our community users. Our newest release, HPCC Systems 8.2.0 Gold contains a number of improvements and features that mean our Cloud Native Platform is now ready to be tested and used in production environments. Use these resources to find out more about our Cloud Native Platform:

Cross platform changes are also included in this blog which are relevant to users of both our Cloud Native and Bare Metal Platforms. This blog includes the main highlights, for the full list of changes, see the HPCC Systems 8.2.0 Release Notes.

Advance notice about HPCC Systems 8.4.0

Going forward, please note that HPCC Systems 8.2.0 is the last release that will include the HPCC Systems Virtual Machine image, which can be used under Oracle VM VirtualBoxTM to bring up a single-node system in a virtual machine. From HPCC Systems 8.4.0, we plan to replace this with a Docker image that can be run locally on Docker Desktop or Minikube, or on most Cloud service providers such as Microsoft Azure, Amazon Web Services, Google Cloud etc. This approach is much more flexible, uses much more modern technology, carries less overheads and uses our latest cloud-native version of HPCC Systems. More details of how to do this will be provided as we approach the release of HPCC Systems 8.4.0. 

Cloud Native Platform Changes

HPCC Systems 8.2.0 Gold includes a number of changes in the helm charts and also in the areas of security and data handling. Those of you who have been waiting for more development work to be completed in the area of data handling, will find the changes made in this release provide an improved experience.

Find out more about File Storage on the HPCC Systems Cloud Native Platform, including migrating from earlier versions, in this blog by Gavin Halliday, Enterprise, Lead Architect, LexisNexis Risk Solutions Group.

System Setup and Helm Chart Usage

In HPCC Systems 8.2.0, some simple changes in the way the values.yaml file must be specified are needed. These changes are enforced by the schema and errors will be produced if an incorrect values files is used. For more information on the sorts of error messages you might see and some examples of how to change a specification in the areas mentioned below, see the changes document available in our Helm GitHub Repository.

Service Changes

Some changes have been made to support annotations on services allowing you to control whether public services are connected to internal subnets or published to the internet. The way that services are specified has been changed and some names of fields within service definitions have also been adjusted so that ROXIE, Sasha and ESP services are more consistent in their syntax.

  • ESP and Sasha services – The settings related to the service have been moved into a new service: section within the component settings.
  • ROXIE services – The name of the setting used to define the external port has been changed to servicePort
  • The public setting on ESP services and the external setting on ROXIE services have been replaced by a new visibility setting specifying a user-settable preset. Pre-populated presets cluster, local and global are provided, but others can be added if needed.
  • Annotations and labels can be specified on any service, or for any visibility preset, or for any component pod.

Startup Probe Changes

Some components may take a while to start up, but should not be added to the relevant Kubernetes load balancer until they have done so.

We use standard Kubernetes startup probes and readiness probes to manage this process. In HPCC Systems 8.2.0 it is possible to override the default settings for the startup probe by setting minStartupTime and maxStartupTime on any component, although ROXIE and Dali are the two that are most likely to need it.

Storage Changes

Previously daliStorage, dataStorage and dllStorage could either refer to an existing plane, or it could implicitly define a storage plane. That implicit plane could be based on a pre-existing persistent volume claim (by setting existingClaim) or ephemeral storage (by setting storageClass and storageSize).

From HPCC Systems 8.2.x, these sections are no longer supported and all storage for data, dlls, landing zones etc. is defined in the list of storage.planes.

Implicit planes that were defined within those now unsupported sections, must now use an explicit plane definition. The plane definitions have been extended to allow ephemeral storage to be defined.

Previously, the default storage plane for a particular storage category could be specified by naming it in the appropriate storage section. For example, storage.dataStorage.plane could name the default data storage plane. From HPCC Systems 8.2.x, the default data storage plane is now the first plane with the corresponding category.

Another notable change is that previously, a storage plane had a labels: attribute to indicate what kind of data was stored on the plane. It was a list which if left blank, defaulted to [ data ]. This has now become a single valued category attribute. In the unlikely event of wanting to have multiple categories stored on the same mounted drive, it is possible to define a plane with a duplicate prefix and storage definition with a different category. Planes also support the subPath attribute to allow a subdirectory to be used within a mount point. 

The default values.yaml and example files have been updated to reflect these changes. Usage examples for defining storage planes in HPCC Systems 8.2.0 are documented in our Helm Repository and documentation on default plane validation is also available.

Here are a few additional changes of note in this area you may be interested to hear about:

Default Data Plane Changes

Previously the storagePlane property could be set on an engine to change the default data plane. This has now been renamed to dataPlane to be consistent with the category name. In future other types of planes will be configurable in the same way (such as spill, dll, temp).

Previously, Sasha services defined their storage plane within a nested storage attribute as shown in this example:

 wu-archiver:    storage:      plane: sasha

In HPCC Systems 8.2.0, this has been simplified as follows:

 wu-archiver:    plane: sasha

A value of “” can be used to indicate that you want to use the first plane with the Sasha category.

General Helm Chart and Cloud Native Platform Changes

The following new features and enhancements have also been added in HPCC Systems 8.2.0:

Cross Platform Changes

The following changes provide new features and enhancements to all our users whether you are using our Bare Metal or Cloud Native platform. The following new features and enhancements are now available in HPCC Systems 8.2.0 Gold:

Security Changes

HPCC Systems components, such as Dali, ESP, eclccserver, dafilesrv, Thor, ROXIE, etc often communicate with each other.  Thor and ROXIE also support internal communication between their own set of processes. For a while HPCC Systems has supported encryption in transit for many of these communication paths, but in HPCC Systems 8.2.0, three important communication paths can also be configured to be secure:

Since HPCC Systems 8.0.0, we now support Jetstack cert-manager. Information about how to use Jetstack cert-manager to automatically provision and manage TLS certificates for the HPCC Systems cloud native platform, is available in this tutorial located in the HPCC Systems Helm Chart Examples GitHub Repository.

For those wanting to use the Hashicorp vault based cert-mananger, installation information is available in this GitHub Repository and also in this tutorial provided by Hashicorp.

For all components sending traffic to and from Dali and also for intra-Thor traffic, the encryption used is standard TLS (Transport Layer Security). This is the same method used in the existing secure communication in HPCC Systems, such as for clients connecting with ESP and components connecting with dafilesrv. For intra-ROXIE traffic, the encryption is AES at the packet level. Look out for a more detailed blog on this subject coming soon, written by Mark Kelly and Anthony Fishbeck.

The following additional security features and enhancements are also included in HPCC Systems 8.2.0 Gold:

New ECL Watch Preview

HPCC Systems 8.2.0. Gold includes a sneak preview (a snapshot taken from Q1 2021) of the new ECL Watch changes that are under development and targeting the end of 2021 for a gold release.

Currently, ECL Watch uses “end of life'” frameworks (Dojo), older JavaScript standards (ES5) and is in need of modernisation. The plan is to port ECL Watch to modern frameworks (React and Fluent UI) and port all JavaScript over to TypeScript, which in turn will allow us to generate modern flavours of JavaScript (ES6, ES2017, ES2020 etc.). Ultimately all the Dojo based pages will be retired.  The roadmap for these changes involves:

  1. Switching from Dojo Build to Webpack (completed in HPCC Systems 7.x.x)
  2. Adding TypeScipt support into the build process (completed in HPCC Systems 7.x.x)
  3. Porting all Dojo pages to React + Fluent UI (this is the bulk of the work that is currently under development)
  4. Re-imagining and improving pages (mostly future work, but some of this work is currently under development)

While the goal is to ensure this modern ECL Watch will be familiar to existing users, it comes with some fundamental changes:

  • Flatter UI.  For example viewing a WU Details page no longer creates an “in page tab”.
  • All hyperlinks are real links (you can bookmark them, right click and open in new page, copy the URL into emails etc.)
  • The address bar URL is more logical and invites the user to manually edit and share it.
  • Forward and Back buttons work as expected.
  • Dark Mode support.

HPCC Systems 8.2.0 Gold includes an early preview of some of this work and can be viewed by navigating to this link.  To give you an idea of how the new pages will look, here are two screenshots showing:

A demo of the the new Workunits page:

Image showing a preview of the new ECL Watch Workunits page

A demo of the new Workunit Details page:

Image showing a preview of the new ECL Watch Workunit Details page

Note:  These images show demo pages within our development test harness and will be subject to change between now and the final released version.

The following new features have been added to the existing ECL Watch in HPCC Systems 8.2.0.Gold:

Documentation Updates

New changes are always accompanied by updated documentation and it is always a good idea to make sure you have the latest version to hand. Use our Containerized HPCC Systems Guide to get started with your cloud based HPCC Systems deployment. Documentation and examples are also available in our Helm Chart Repository and our documentation team have provided some “How To” videos to help you get started.

All documentation for our Cloud Native and Bare Metal Platforms is available on our website along with the release notes for HPCC Systems 8.2.0. Gold and our Online Training Classes. Bare Metal users will find the new version available on the Downloads page.

Here are the documentation update highlights available in HPCC Systems 8.2.0 Gold:

Containerized HPCC Systems Guide

Find out more about the new instructions for using hpcc-localfile and the new drop zone syntax mentioned in the Systems Setup and Helm Chart Usage section above.

ECL Language Reference Guide

Note that there are changes to the Basic Constants section.

Using ECL Watch Guide

Find out more about the new button next to the copy the WUID to clipboard button, allowing you to copy a direct link of the widget to the clipboard.

How To Videos

See the full list of available How To Videos. The newest available videos are:

HPCC Systems GitHub Helm Chart Repository Documentation and Examples

View the following changes in the values.yaml file:

  • Adding a drop zone to example values.yaml files
  • Annotation on values.yaml
  • Allow environment variable to be defined in values.yaml

Changes mentioned in sections above mention the following examples files and documentation available in the HPCC Systems Helm Chart GitHub Repository:

Tags
Cloud
Community Content
Features & Tools