Features new and improved in HPCC Systems 8.2.0 Gold
Earlier in 2021, we announced the arrival of HPCC Systems 8.0.0 Gold, which heralded the release of our new Cloud Native Platform, which was made available for Cloud performance evaluation by our community users. Our newest release, HPCC Systems 8.2.0 Gold contains a number of improvements and features that mean our Cloud Native Platform is now ready to be tested and used in production environments. Use these resources to find out more about our Cloud Native Platform:
- HPCC Systems Cloud Native Wiki Page – Provides access to blogs, how to videos and links to various GitHub repository resources
- HPCC Systems 8.0.0 – Cloud Native Platform Highlights -Find out about getting started with our new platform and including Helm Chart usage, security and data handling
- HPCC Systems Helm GitHub Repository – For deploying HPCC Systems under Kubernetes
- Supporting Documentation – Containerized HPCC Systems Platform
Cross platform changes are also included in this blog which are relevant to users of both our Cloud Native and Bare Metal Platforms. This blog includes the main highlights, for the full list of changes, see the HPCC Systems 8.2.0 Release Notes.
Advance notice about HPCC Systems 8.4.0
Going forward, please note that HPCC Systems 8.2.0 is the last release that will include the HPCC Systems Virtual Machine image, which can be used under Oracle VM VirtualBoxTM to bring up a single-node system in a virtual machine. From HPCC Systems 8.4.0, we plan to replace this with a Docker image that can be run locally on Docker Desktop or Minikube, or on most Cloud service providers such as Microsoft Azure, Amazon Web Services, Google Cloud etc. This approach is much more flexible, uses much more modern technology, carries less overheads and uses our latest cloud-native version of HPCC Systems. More details of how to do this will be provided as we approach the release of HPCC Systems 8.4.0.
Cloud Native Platform Changes
HPCC Systems 8.2.0 Gold includes a number of changes in the helm charts and also in the areas of security and data handling. Those of you who have been waiting for more development work to be completed in the area of data handling, will find the changes made in this release provide an improved experience.
Find out more about File Storage on the HPCC Systems Cloud Native Platform, including migrating from earlier versions, in this blog by Gavin Halliday, Enterprise, Lead Architect, LexisNexis Risk Solutions Group.
System Setup and Helm Chart Usage
In HPCC Systems 8.2.0, some simple changes in the way the values.yaml file must be specified are needed. These changes are enforced by the schema and errors will be produced if an incorrect values files is used. For more information on the sorts of error messages you might see and some examples of how to change a specification in the areas mentioned below, see the changes document available in our Helm GitHub Repository.
Service Changes
Some changes have been made to support annotations on services allowing you to control whether public services are connected to internal subnets or published to the internet. The way that services are specified has been changed and some names of fields within service definitions have also been adjusted so that ROXIE, Sasha and ESP services are more consistent in their syntax.
- ESP and Sasha services – The settings related to the service have been moved into a new service: section within the component settings.
- ROXIE services – The name of the setting used to define the external port has been changed to servicePort
- The public setting on ESP services and the external setting on ROXIE services have been replaced by a new visibility setting specifying a user-settable preset. Pre-populated presets cluster, local and global are provided, but others can be added if needed.
- Annotations and labels can be specified on any service, or for any visibility preset, or for any component pod.
Startup Probe Changes
Some components may take a while to start up, but should not be added to the relevant Kubernetes load balancer until they have done so.
We use standard Kubernetes startup probes and readiness probes to manage this process. In HPCC Systems 8.2.0 it is possible to override the default settings for the startup probe by setting minStartupTime and maxStartupTime on any component, although ROXIE and Dali are the two that are most likely to need it.
Storage Changes
Previously daliStorage, dataStorage and dllStorage could either refer to an existing plane, or it could implicitly define a storage plane. That implicit plane could be based on a pre-existing persistent volume claim (by setting existingClaim) or ephemeral storage (by setting storageClass and storageSize).
From HPCC Systems 8.2.x, these sections are no longer supported and all storage for data, dlls, landing zones etc. is defined in the list of storage.planes.
Implicit planes that were defined within those now unsupported sections, must now use an explicit plane definition. The plane definitions have been extended to allow ephemeral storage to be defined.
Previously, the default storage plane for a particular storage category could be specified by naming it in the appropriate storage section. For example, storage.dataStorage.plane could name the default data storage plane. From HPCC Systems 8.2.x, the default data storage plane is now the first plane with the corresponding category.
Another notable change is that previously, a storage plane had a labels: attribute to indicate what kind of data was stored on the plane. It was a list which if left blank, defaulted to [ data ]. This has now become a single valued category attribute. In the unlikely event of wanting to have multiple categories stored on the same mounted drive, it is possible to define a plane with a duplicate prefix and storage definition with a different category. Planes also support the subPath attribute to allow a subdirectory to be used within a mount point.
The default values.yaml and example files have been updated to reflect these changes. Usage examples for defining storage planes in HPCC Systems 8.2.0 are documented in our Helm Repository and documentation on default plane validation is also available.
Here are a few additional changes of note in this area you may be interested to hear about:
- A new example of azure file storage generated from a storage account is now available
- Spray functionality has been improved and fsGetDropZones() and fsGetDefaultDropZone() have been implemented
- There is a new option for specifying the number of file parts for the file services spray function
- The syntax for spraying from a Kubernetes landing zone has been improved
- You can now add a drop zone as a storage plane. An example of how to do this is provided in the stock values.yaml file.
- Planes are now able to create implicit persistent volume claims
- A warning is now generated on a Helm install if any ‘in use’ plane is using ephemeral storage
- Use planes for ephemeral storage
- A default data plane is now generated
Default Data Plane Changes
Previously the storagePlane property could be set on an engine to change the default data plane. This has now been renamed to dataPlane to be consistent with the category name. In future other types of planes will be configurable in the same way (such as spill, dll, temp).
Previously, Sasha services defined their storage plane within a nested storage attribute as shown in this example:
wu-archiver: storage: plane: sasha
In HPCC Systems 8.2.0, this has been simplified as follows:
wu-archiver: plane: sasha
A value of “” can be used to indicate that you want to use the first plane with the Sasha category.
General Helm Chart and Cloud Native Platform Changes
The following new features and enhancements have also been added in HPCC Systems 8.2.0:
- Environment variables can now be defined in Helm Chart settings
- Error reporting on kubectl failure has been improved
- The default image version now matches the helm chart version and a helm chart version label for each pod is created showing when the helm chart was tagged and published to make troubleshooting easier.
- Helm chart functionality for setting up a cert-manager and or checking secrets dependencies can now be turned off. This is needed when running in template mode which can’t talk to the Kubernetes API services
- An auto detect config file updates and reload mechanism has been added
- A JobID field has been aded to the default log layout
- Publishing files to ROXIE in container mode has been improved
- The ESP ws_machine service has been removed from the container build
- Standard log columns have been updated in container mode
- Unnecessary pod restarts on configMap changes are now prevented
- The kubectl version in Dockerfile has been upgraded to the latest release (1.18.18)
- ‘pods:[“all”]’ has been added for placements assignment which will apply the placement for all HPCC Systems pods/jobs.
- internal/expert options can now be set via the helm values
- WsFileIO.CheckServerAccess has now been implemented in container mode
- An option has been added to Thor to allow file parts to be written to individual subdirectories
- Containerized ROXIE now copies files once rather than multiple times
- Loading agent queries in ROXIE has been improved
- An issue with ROXIE restarting during startup if file copies take too long has been resolved
- The GNN docker image has been upgraded to Tensorflow 2.5 and the latest Nvidia CUDA drivers
- A logging version of Prometheus metric sink is now available
Cross Platform Changes
The following changes provide new features and enhancements to all our users whether you are using our Bare Metal or Cloud Native platform. The following new features and enhancements are now available in HPCC Systems 8.2.0 Gold:
- Additional metrics have been added to ESP, including Count of SOAP requests and Active request count
- An experimental generic disk read activity has been implemented in hThor
- ROXIE UDP layer now handles a server disappearing or restarting more efficiently
- An issue with the potential loss of lines reading super CSV has been resolved
- New SOAPCALL option to log text at the end of a SOAP request
- Ambiguous ‘–name’ parameters on the ECL command line have been improved
- TLS Support in Thor is now available – For more information see Security Changes below.
Security Changes
HPCC Systems components, such as Dali, ESP, eclccserver, dafilesrv, Thor, ROXIE, etc often communicate with each other. Thor and ROXIE also support internal communication between their own set of processes. For a while HPCC Systems has supported encryption in transit for many of these communication paths, but in HPCC Systems 8.2.0, three important communication paths can also be configured to be secure:
Since HPCC Systems 8.0.0, we now support Jetstack cert-manager. Information about how to use Jetstack cert-manager to automatically provision and manage TLS certificates for the HPCC Systems cloud native platform, is available in this tutorial located in the HPCC Systems Helm Chart Examples GitHub Repository.
For those wanting to use the Hashicorp vault based cert-mananger, installation information is available in this GitHub Repository and also in this tutorial provided by Hashicorp.
For all components sending traffic to and from Dali and also for intra-Thor traffic, the encryption used is standard TLS (Transport Layer Security). This is the same method used in the existing secure communication in HPCC Systems, such as for clients connecting with ESP and components connecting with dafilesrv. For intra-ROXIE traffic, the encryption is AES at the packet level. Look out for a more detailed blog on this subject coming soon, written by Mark Kelly and Anthony Fishbeck.
The following additional security features and enhancements are also included in HPCC Systems 8.2.0 Gold:
- A password is now required for LDAP users created via the ECL Watch user interface
- Multiple LDAP addresses in the LDAP configuration can now be comma separated
New ECL Watch Preview
HPCC Systems 8.2.0. Gold includes a sneak preview (a snapshot taken from Q1 2021) of the new ECL Watch changes that are under development and targeting the end of 2021 for a gold release.
Currently, ECL Watch uses “end of life'” frameworks (Dojo), older JavaScript standards (ES5) and is in need of modernisation. The plan is to port ECL Watch to modern frameworks (React and Fluent UI) and port all JavaScript over to TypeScript, which in turn will allow us to generate modern flavours of JavaScript (ES6, ES2017, ES2020 etc.). Ultimately all the Dojo based pages will be retired. The roadmap for these changes involves:
- Switching from Dojo Build to Webpack (completed in HPCC Systems 7.x.x)
- Adding TypeScipt support into the build process (completed in HPCC Systems 7.x.x)
- Porting all Dojo pages to React + Fluent UI (this is the bulk of the work that is currently under development)
- Re-imagining and improving pages (mostly future work, but some of this work is currently under development)
While the goal is to ensure this modern ECL Watch will be familiar to existing users, it comes with some fundamental changes:
- Flatter UI. For example viewing a WU Details page no longer creates an “in page tab”.
- All hyperlinks are real links (you can bookmark them, right click and open in new page, copy the URL into emails etc.)
- The address bar URL is more logical and invites the user to manually edit and share it.
- Forward and Back buttons work as expected.
- Dark Mode support.
HPCC Systems 8.2.0 Gold includes an early preview of some of this work and can be viewed by navigating to this link. To give you an idea of how the new pages will look, here are two screenshots showing:
A demo of the the new Workunits page:
A demo of the new Workunit Details page:
Note: These images show demo pages within our development test harness and will be subject to change between now and the final released version.
The following new features have been added to the existing ECL Watch in HPCC Systems 8.2.0.Gold:
- The workunit complete icon has been darkened which distinguishes it more clearly from the running workunit icon.
- The ROXIE cluster column has been removed from the ECL workunit list page since it is no longer needed
Documentation Updates
New changes are always accompanied by updated documentation and it is always a good idea to make sure you have the latest version to hand. Use our Containerized HPCC Systems Guide to get started with your cloud based HPCC Systems deployment. Documentation and examples are also available in our Helm Chart Repository and our documentation team have provided some “How To” videos to help you get started.
All documentation for our Cloud Native and Bare Metal Platforms is available on our website along with the release notes for HPCC Systems 8.2.0. Gold and our Online Training Classes. Bare Metal users will find the new version available on the Downloads page.
Here are the documentation update highlights available in HPCC Systems 8.2.0 Gold:
Containerized HPCC Systems Guide
Find out more about the new instructions for using hpcc-localfile and the new drop zone syntax mentioned in the Systems Setup and Helm Chart Usage section above.
ECL Language Reference Guide
Note that there are changes to the Basic Constants section.
Using ECL Watch Guide
Find out more about the new button next to the copy the WUID to clipboard button, allowing you to copy a direct link of the widget to the clipboard.
How To Videos
See the full list of available How To Videos. The newest available videos are:
- Setting up Persistent Storage on an HPCC Systems Containerized Deployment
- Using ECL Watch Advanced Search
- Creating a PackageMap File
HPCC Systems GitHub Helm Chart Repository Documentation and Examples
View the following changes in the values.yaml file:
- Adding a drop zone to example values.yaml files
- Annotation on values.yaml
- Allow environment variable to be defined in values.yaml
Changes mentioned in sections above mention the following examples files and documentation available in the HPCC Systems Helm Chart GitHub Repository:
- Storage changes in values-auto-azurefile.yaml and values-retained-azurefile.yaml
- Security changes relating to Jetstack Cert-Manager and Hashicorp Vault Based Cert Manager