The Download: Tech Talks by HPCC Systems – An Interview with Shamser Ahmed
Welcome to the fifth installment of the summer series of The Download: Tech Talks by HPCC Systems
This series focuses on information about the latest developments within HPCC Systems related to the cloud. HPCC Systems has been very active with projects over the past few months, getting better support for dynamic environments via container orchestration and other capabilities.
The fifth installment of the summer series of The Download: Tech Talks by HPCC Systems, features an interview with Shamser Ahmed (Senior Consulting Software Engineer in HPCC Systems Development at LexisNexis Risk Solutions), and Flavio Villanustre (VP and Chief Security Officer, LexisNexis Risk Solutions).
The full recording of Flavio and Shamser’s interview is available on the HPCC Systems YouTube channel.
The audio version is available on Soundcloud.
Meet our presenters
Flavio Villanustre – Flavio Villanustre is CISO and VP of Technology for LexisNexis® Risk Solutions. He also leads the open source HPCC Systems platform initiative, which is focused on expanding the community gathering around the HPCC Systems Big Data platform, originally developed by LexisNexis Risk Solutions in 2001 and later released under an open source license in 2011. Flavio’s expertise covers a broad range of subjects, including hardware and systems, software engineering, and data analytics and machine learning. He has been involved with open source software for more than two decades, founding the first Linux users’ group in Buenos Aires in 1994.
Shamser Ahmed – Shamser Ahmed is a Senior Consulting Software Engineer in HPCC Systems Development, at LexisNexis Risk Solutions. He joined the HPCC Systems Platform team in 2015, having spent 4 years as the tech lead for the ESP team in the UK. Shamser is involved in many different areas including the code generator, the analyzer, and changes to support the cloud.
This blog provides a brief summary of topics discussed in the interview, including:
- Latest Developments
- Cloud Provisioning
- Workunit Analysis Tool
- Additional Information
Let’s begin by discussing the latest developments in the HPCC Systems “Journey to the Cloud.”
Latest Developments
The HPCC Systems cloud development team continues to make significant progress on the new “cloud native” platform. The new operating environment is comprised of Docker containers managed by Kubernetes, along with continued support for “bare metal” installations. Helm charts are used to deploy these containers on cloud platforms that support Kubernetes.
It is now possible to run an HPCC systems cluster inside an Amazon Web Services (AWS) Elastic Cloud (EC2). In addition to flexibility and unlimited computing capacity, running the HPCC Systems platform on the cloud allows for:
- Proof-of-concept
- Experimentation
- Learning
- Leveraging the platforming without incurring costs for hardware and administration
- Creating and using the HPCC Systems cluster without buying and installing new hardware.
The latest HPCC Systems version is available for download on the website. The “cloud native” platform is not yet production-ready, but version 8.0, which is the “production” version, will be released in 2021.
One of the major considerations when running jobs on the cloud is cost. Let’s take a look at how HPCC Systems plans to address cost when working in the cloud.
Cloud Provisioning
The HPCC Systems cloud development team is currently developing the “cloud provisioning” component of the “cloud native” platform. Cloud provisioning is the allocation of resources and services from a cloud provider to the customer. The benefits of cloud provisioning include scalability, speed, and potential cost savings.
The goal of HPCC Systems for the cloud provisioning component is to provide information about how much a job will cost to execute in a cloud environment. Cost factors include pre-execution time, processing time, storage, and data transmission. Knowing the cost of a job based on cluster size, speed, and other activities within the job will allow the customer to decide the best option.
In the first phase of cloud provisioning, an estimate of the cost for executing a job will be provided in ECL Watch. ECL Watch is a component of the HPCC Systems platform that allows the user to monitor jobs. Cost will be provided as part of the Helm configuration.
The present challenge is getting cost information from the cloud provider and providing the information automatically. Currently, the user manually changes the configuration.
Now, let’s take a look at how a recently-introduced tool will be used in the HPCC Systems cloud native environment.
Workunit Analysis Tool
In HPCC Systems version 7.6.x, the Workunit Analysis tool was introduced. This tool is used to analyze workunits and provide vital information about performance. For more information on this tool, see the blog, “Optimise workunit performance using the HPCC Systems Workunit Analysis Tool,” by Shamser Ahmed.
Future plans include using the workunit analyzer to examine peak memory usage, storage, types of activities, data transfers, and other factors for an entire job, and employing that information to advise on the best architecture to achieve the fastest execution time or lowest cost in the cloud environment.
For more information about HPCC Systems and the “journey to the Cloud” see the information below.
Additional Information
Videos
The following videos provide additional details about the HPCC Systems cloud development project:
“The Download, Tech Talks by HPCC Systems: An Interview with Shamser Ahmed” – available on the HPCC Systems YouTube channel, under 2020 Tech Talks.
“The Download, Tech Talks by HPCC Systems: An Interview with Ming Wang” – available on the HPCC Systems YouTube channel, under 2020 Tech Talks.
“The Download, Tech Talks by HPCC Systems: An Interview with Gavin Halliday” – available on the HPCC Systems YouTube channel, under 2020 Tech Talks.
“The Download, Tech Talks by HPCC Systems: An Interview with Jake Smith” – available on the HPCC Systems YouTube channel, under 2020 Tech Talks.
Blogs
More information about HPCC Systems and the cloud can be found in the following blogs:
“The Download, Tech Talks by HPCC Systems: An Interview with Richard Chapman” – This is the first installment of the summer series of The Download: Tech Talks by HPCC Systems, featuring an interview with Richard Chapman (VP Technology, HPCC Systems Development at LexisNexis Risk Solutions), and Flavio Villanustre (VP and Chief Security Officer, LexisNexis Risk Solutions).
“HPCC Systems and the Path to the Cloud,” by Richard Chapman: In this blog, Richard Chapman talks about the “journey to the Cloud” and demonstrates how to set up a simple test cluster using a default Helm Chart.
“Setting up a default HPCC Systems cluster on Microsoft Azure Cloud Using HPCC Systems 7.8.x and Kubernetes,” by Jake Smith: This blog outlines the steps required to set up and run the HPCC Systems Helm Charts using the Azure Kubernetes Service (AKS).
“Persisting data in an HPCC Systems Cloud native environment,” by Gavin Halliday: This blog covers how to persist your data in an HPCC Systems cloud native environment.
Acknowledgements
A special thank you to Shamser Ahmed and Flavio Villanustre for this informative update about the HPCC Systems cloud development project.