Fly on the Wall – Developer Offsite Meeting 2020

These developer offsite meetings have been a regular feature of the team’s working landscape for 6 years. The aim is to collectively discuss ongoing development and strategic issues relating to the future of the HPCC Systems open source project. Since the majority of the HPCC Systems core platform team generally work remotely from each other, it is a rare opportunity to get everyone in the same room.

The design discussion meetings are detailed and intense, which means avoiding day to day distractions as much as possible, including email! Everyone has contributed to the agenda and some definite themes have emerged including HPCC Systems cloud support, more security enhancements, improved usability and the machine learning library. Changes in each area have the potential to have a wide impact so everyone’s views are needed.

It’s a week long event, which always raises some curiosity about what it must be like to be there from those who have not experienced it. When I am asked this question, I’m never quite sure how to reply. I guess there could be an air of mystery attached to the disappearance of the developers to a remote location while being noticeable by their absence on email. Where have they gone and what are they doing?

It’s tempting to paint a scene of a group of developers holed up in a cave, or perhaps gathered in tents around a campfire in the middle of nowhere with only rudimentary facilities, very little privacy and living off smores! It’s not an altogether bad image really, although I guess not everyone is a happy camper and although I love smores, more than two in quick succession is usually followed by regret in my experience.

The chosen venue has to house around 12 people. I always chuckle a little that even the largest short terms rentals may be found on a website for holiday ‘cottages’, which for me, conjurs up an image of a small and cosy dwelling, sort of like the one in which Kate Winslet’s character lives in the movie ‘The Holiday’.

In reality, while their home for the week, The Oaks, is undoubtably very cosy and comfortable, it is also sizeable, with ensuite bedrooms and several reception rooms, all of which are large enough to seat everyone. There’s also large kitchen fitted with multiple hobs, ovens, fridges, freezers and dishwashers. This becomes important later!

The remote location chosen would usually also mean fabulous views from numerous rooms. This is great for daydreamers like me who sort out their thoughts and get inspiration by disappearing temporarily through the window. However, the weather in the UK’s Lake District was awful all week (I guess the hint is in the name), so the ‘visionaries’ amongst us had to be content looking out on to the pleasant garden, imagining what lay beyond the mist.

We are well in to the HPCC Systems 7 series of releases so the focus of this meeting is very much what lies further ahead. In fact, the main discussion points revolve around what needs to be done for HPCC Systems to be fully Cloud supportive. HPCC Systems already supports the Cloud and has done for some time. But the aim is to move towards providing the same level of support that we do on bare metal.

A Cloud native HPCC Systems will provide the means to run as many Thors as needed at any one time, scaling up and down the number of nodes on the fly to meet the load. Access to GPUs will also be readily available which helps, for example, when using HPCC Systems with Tensorflow, which can take advantage of both CPU and GPU.

Given that this topic dominated the discussions, it was no surprise to me to hear that our colleague, Jon Burger (Senior Architect, LexisNexis Risk Solutions), joined the team earlier in the week to contribute to the discussion on the planned development work. The notes from the meeting are peppered with comments showing how valuable it was for Jon to be there to share his knowledge and insights with the HPCC Systems platform developers. Incidentally, those of you who attend our Tech Talk webcast series will remember that Jon presented about Hive 360 back in May 2017, speaking about leveraging the on-demand benefits of Infrastructure as a Service (IaaS) cloud technology.

The Road to the Cloud

I’ve already said that HPCC Systems can already be used in the Cloud so I’m interested to find out what the planned development work involves and how it will improve on what is already available. I ask the question and the reply is that the current support is more of a ‘lift and shift’ approach with the aim being to make it a natural fit, removing some of the obstacles which can currently make running HPCC Systems on the cloud a little complex, allowing users to take full advantage of the benefits of a modern cloud infrastructure. This means making a number of changes but particularly moving away from:

• Using local storage that comes for free on hardware.
• Using static IP addresses.

Since in a Cloud environment the machines that are being used for your HPCC Systems environment can change,  it cannot be used for long term storage and the IP addresses cannot be relied on as persistent identifiers unless, of course, the environment was to be kept up and running 24/7, which defeats the object of using the Cloud and is an expensive option!

Some intial work has already been completed to pave the way for the main body of work involved. For example, hostnames are now resolved to IP addresses much later in the startup process, but there is much more to do. Richard Chapman is writing a blog on HPCC Systems and the Cloud and I don’t want to steal his thunder, so for the purposes of this blog, I’m going to give you a taste of what is coming in the immediate future. The main focus of development at the time of writing is on producing a proof of concept that reinforces the direction of travel of the current development work.

^{Developers at work}

Here is what you can expect to see:

• Containerized architecture, touch free with one HPCC Systems process per container which starts when the container starts and stops when it stops, keeping no internal state.
• Configuration of each component controlled via separate standardised configuration files that can be integrated into an external cluster management system like Kubernetes/Helm in a way that is familiar to any K8s/Helm user.
• Support off node storage for Thor and ROXIE data as well as Dali metadata.
• Remove requirements for HPCC Systems to be told cluster topology in advance.
• Support on-demand scaling.
• Treat K8s as our target platform allowing us to maintain cloud neutrality.

HPCC Systems 7.8.0, will be the first release to include some of these initial changes, giving us the opportunity to test our assumptions while we continue to work towards achieving our overall aims in the releases to follow.

I usually arrive at these offsite meetings on the penultimate day and to help me catch up, the developers review and consolidate the discussions that have happened during the week. On this occasion, I listened while adding in the JIRA issues needed to complete the work which required some fast typing! As the parent issues were added, suggestions for sub-tasks were coming thick and fast as they worked through what was needed to complete each item. I could tell that the team was feeling excited and a little impatient to see their discussions come to life and that excitement was a little infectious. Use this JIRA filter to see the full list of planned changes in this area.

We celebrated the 20 year anniversary of HPCC Systems at our Community Day Summit in 2019, when Flavio Villanustre spoke about how our platform started and how it has responded to changes in technology and the demands of our data driven world (Watch Recording / View Slides).

I can’t help thinking that we will be looking back on these improvements in years to come as another of those significant moments of change.

Other development highlights

While the move to the cloud dominated this offsite meeting, it was not the only topic that was discussed. Here are some other highlights to look out for:

Workunit Analysis Tool

This tool is new in the HPCC Systems 7.6x series. To find out what is available now, read Shamser Ahmed’s blog about optimising workunit performance. A number of improvements are planned for this tool:

• Improving the user interface in ECL Watch to highlight workunits which have generated specific warnings.
• Collecting warnings in a log file that can be used by third party tools to send alerts to users.
• Progressive analysis, to allow users to terminate jobs that are reporting issues indicating that the workunit would for example, run for too long or would generate incorrect results.
• Take users to the location of the problem in the graph and show information explaining the problem.
• New rules to be added that extend the range of problems that can be reported, such as skewed spills, issues caused by sequential operation, time spent in functions and soap calls, index read/keyed joins and a large number of reject rows.

The plan is to keep adding new features to this tool, which runs automatically after each Thor job, saving ECL developers from a potentially long manual analysis process.

Log Visualizer

This tool was introduced in HPCC Systems 7.0.0. Rodrigo Pastrana presented a short demo of this tool in action to demonstrate the power of visualised log data using the Elastic stack integration. Users can create dashboards to monitor for example, system behaviour and activity using the information contained in the log files. In fact, the possibilities of what can be monitored using the Log Visualizer are wide and varied. For example, Rodrigo showed how to monitor user interaction with HPCC Systems using the ECL Watch User/Session Management data. He also showed how to track query health and identify opportunities for ECL optimisation by monitoring Roxie query usage, looking at the performance analytics and tracking information.

It was a great demo that got the developers thinking about metrics that might be helpful to add as ‘canned’ dashboards for common use cases. But the message really is that the possibilities are endless and dashboards can be customised to visualise metrics that are relevant to specific user needs. Read Rodrigo’s blog to find out how to integrate your ELK based visualizations into ECL Watch using the HPCC Systems Configuration Manager and see some examples of log visualizations.

ECL Watch and ECL extension for VS Code

Various user interface improvements are planned for ECL Watch including better integration of user preferences a faster graph viewer and an improved ECL archive viewer.

In terms of ECL language support in third party environments, we already support VS Code and continued development is planned to improve support for:

• ECL language
• Client Tools
• Bundles
• Auto formatting

Find out more about the ECL extension for VS Code, including installation and settings in the Visual Studio Marketplace.

Security

The work to be done in this area focuses on what is needed to support the planned cloud development work, although some may also be relevant to our bare metal platform. Look out for:

• Support for component level mutually authenticated TLS.
• Automated component certificate generation using the built-in Kubernetes or other certificate authority.
• Support for Kubernetes secret objects for container level configured secure secrets.
• Integration of secret managers for managing keys, credentials, and other highly sensitive information at the ECL, components, and services level.
• Defaulting EclWatch and other services to use HTTPS.
• Integration of cloud based directory services such as Azure AD and eventually federated services.

Dynamic ESDL

In HPCC Systems 7.0.0, some major changes were introduced to make instantiating and managing ESDL services at run time more dynamic. In his blog Dynamic ESDL is more dynamic in HPCC Systems 7.0.0, Yanrui Ma walks through these changes using a simple demonstration. More improvements are planned in this area including:

• A new scripting engine to make the application of integration, security, and transformation logic without writing service Java or C++ code will help to streamline the release cycle as services are developed and continue to evolve.
• A “config as code” ESDL configuration bundling tool. This will make it easy to create manageable release bundles for dynamic services.
• A new container based ESDL service configuration, management and deployment model.

Supporting documentation on using Dynamic ESDL is available on our website.

Machine Learning

Roger Dev, who leads the HPCC Systems Machine Learning Library project, gave a quick update on the new additions we can expect to see later in the year. The Generalized Neural Network (GNN) bundle mentioned below is new to our Machine Learning Library. Roger has written a blog about analysing images, videos and time series using the new GNN bundle and the bundle itself can be found in the GNN GitHub repository.

• Generalised Neural Network (GNN) bundle version 2, providing fully flexible Keras/Tensorflow support. This includes support for Functional API (non-sequential models), multiple models within a work unit and Tensorflow 2.0 support.
• Latent Dirichlet Allocation (LDA) bundle LDA, including support for topic modelling for text.
• Dimensionality Reduction, including Principal Component Analysis for reducing the dimensionality of a dataset.
• Logistic Regression bundle version 2, including multinomial classification

There are a number of blogs available about the HPCC Systems Machine Learning library bundles. Browse the list of blogs by Roger Dev and also the machine learning blog posts written by Lili Xu.

Evenings and weekends

If someone asks how they are going to fit so much work in to so little time, it can be a bit of a standing joke to say well there’s always evenings and weekends. The offsite does indeed involve being away for evenings and a weekend, which makes it important to factor in some downtime. The team did manage to find a break in the awful weather to enjoy a little bit of the lakeland countryside and take in some fresh air.

From the left: Yanrui Ma, Tony Fishbeck, Rodrigo Pastrana, Attila Vamos, Gordon Smith, Jake Smith, Mark Kelly,
Roger Dev, Gavin Halliday and Richard Chapman
Behind the camera: Shamser Ahmed

It has also become tradition that the main ‘extra curricular’ team building exercise involves cooking the evening meal. Hence my earlier comment about the importance of the quality and number of kitchen appliances!

The team is divided in to small groups and each one chooses a theme. The first team, Gavin Halliday, Mark Kelly, Attila Vamos and Gordon Smith, provided an Italian themed menu.

I arrived prepared to cook a Fajita Bar menu with Roger Dev, Jake Smith and Yanrui Ma. Although, I was told as soon as I arrived that it would be hard to beat the vegetable pad thai produced the previous evening by Richard Chapman, Shamser Ahmed, Rodrigo Pastrana and Tony Fishbeck. I do like a challenge!!

It’s a lot of fun. It’s also a very good team building exercise. Not everyone wants to be a chef but there’s always the equally important sous chef role and then there’s the go/no go stand up meeting, with the hope that the end result delivers the promise anticipated by the rest of team. Not so different from the day job really!

HPCC Systems Developer Restaurant: Left row: Gordon Smith, Rodrigo Pastrana, Roger Dev, Jake Smith, Jon Burger
Right row: Attila Vamos, Mark Kelly, Gavin Halliday, Yanrui Ma, Tony Fishbeck, Shamser Ahmed

I wasn’t there earlier in the week so I can’t say whether the previous dinner preparations were more like Hell’s Kitchen or MasterChef. The team I joined was definitely more like MasterChef!