What's new in HPCC Systems 7.10.0 Gold
HPCC Systems is going cloud native and since 7.8.0 Gold, users have been able to take the initial stages of this development work for a test drive.
HPCC Systems 7.10.0 Gold, builds on this cloud native development work as well as providing some additional features that are equally as applicable to our bare metal version. So whichever type of platform you are using there is something for you here.
HPCC Systems Cloud Native Platform
There are a number of blogs available which take you through our development plans for this version of the platform and provide tutorial style instructions to help you get started on various cloud providers:
- Path to the Cloud
Find out about development plans and how to get started by setting up a default HPCC Systems cluster using a Helm chart. - Setting up a Default Cluster on Microsoft Azure using Kubernetes
A tutorial style blog providing the steps required and additional notes. - Persisting your data
Includes instructions on how to do this and provides a temporary solution for how to get your data on to an HPCC Systems cluster while we work on the permanent solution. - Cloud Native Wiki Page
All the resources currently available to help early adopters.
The following features are now available in the cloud native version of HPCC Systems 7.10.0 Gold.
Adding and Persisting Data
We have started to add some support for Azure Blobs. This development work is in the early stages and there is much more to come in the near future.
By 7.12.0, the plan is to allow a lot more flexibility on where data is stored, including Azure blobs, Azure Data Lake and Amazon S3 buckets. However, we also want to allow backward compatibility and interoperability with existing bare metal HPCC Systems environments.
To find out more about the planned development in this area and what has already been completed see the following JIRA issues:
Configuration
- Helm files are now available for local storage, AWS EKS and Microsoft Azure There is also a general NFS solution for persisting data, which has been tested on GKE, but may also work with other cloud platforms.
- We now support an Ubuntu 20.04 container image which we are now using as the base image for our cloud containers.
- We are now using a single container image for all components rather than a specific one for each component.
- It is much easier to run a default configuration of ECL Watch and other pre-canned configurations of ESP services.
- There is now an option to specify the number of Thor slaves per job in a cloud setup.
Security
- We now support Kubernetes secrets.
- We now support Azure Active Directory.
- LDAP Security is now able to set and retrieve permissions in Azure Active Directory.
Usability
- Bundles can now be named in the Helm chart, which simplifies the process for installing a bundle significantly.
Our cloud native HPCC Systems platform is an ongoing development project. While it isn’t ready for production environments, it is available for you to take a look at how it works, give it a try and contribute to the development process by providing us with some feedback about your experience using our Community Issue Tracker.
General Platform Features and Enhancements in HPCC Systems 7.10.0 Gold
The features and enhancements mentioned in this section are available to users of both our bare metal and cloud native platform versions. Of course, this blog includes the edited highlights only. To see the full list of changes, take a look at the release notes.
Early adopters of our cloud native version can get started setting up a cluster using the blogs shown above.
Our bare metal users can download HPCC Systems 7.10.0 Gold from our website.
DESDL Scripting Enhancements
Major enhancements have been made to the DynamicESDL transformation/integration language. The scripting language now supports explicit parameters and scoped variables, control statements such as:
- for-each loops and choose with multiple when clauses.
- The ability to do additional authorization based on contents of the request.
- The ability to abort the transaction.
- More ways of modifying the request in flight, and more.
More details are available in these JIRA issues:
Security
- The STD.Crypto library has been enhanced with four new modules that allow a passphrase to be specified in an ECL DATA parameter:
- SymmEncryption
- PKEncryption
- PKEncryptionFromLFN
- PKEncryptionFromBuffer
With this enhancement, passphrases can now contain a binary blob, complete with embedded NULL characters.
The Cryptographic Standard Library ECL Module was a new edition to the HPCC Systems platform in version 7.6.x. It provides an assortment of cryptographic features to help ECL developers safeguard sensitive data, using industry standards cryptographic algorithms. More information is available in this blog.
- Code signing functionality is now available in ecl cmd allowing you to sign your ECL code with the ECL command line tool.
Usability
- Translation of the HPCC Systems platform is now available in French
- A couple of ECL IDE UI improvements are included in this release:
- Bookmarks on the ribbon bar
- Right click, copy and paste when clicking on an attribute in a repository
- ECL Syntax Highlighting is now available in GitLab. While syntax highlighting has been included as a feature in the platform for some time, it has only recently been supported by GitLab. Find out how this feature will help ECL users to work faster and more efficiently.
- Ability to validate a non active packagemap. This is a feature requested by a user who wanted to be able to validate the contents of a packagemap (as a sanity check) before adding it to the system.
- Add SOAPCALL information to the ECL watch query page. Users can now find out information about the gateways a query is using.
- MySQL integer fields can now map to ECL decimals
- Support for function timings to Thor. This feature reports the time spent calling each external function allowing you to analyse why a query maybe running slower than expected.
Performance
- Parallel load ROXIE queries. This feature means that ROXIE starts up faster because more than one can now be loaded at the same time.
- Default compression has been changed from LZW to LZ4 for Thor pub files which is a much faster compression algorithm.
- Gullotine thor jobs exceeding cost limit. Allows you to set a cost limit which causes Thor jobs to be terminated when that limit is reached.
What’s next…
As you start using this release, we have already moved on to the next! HPCC Systems 7.12.0 will focus on providing a more complete cloud offering including stronger ROXIE cloud support and more storage options. This release is expected to be available in the Fall.