All Documentation

All HPCC Systems Docs

This link provides a single Zip file containing all of the PDF documentation.

Containerized HPCC Systems Platform

This book focuses on containerized deployments. The first section is about using Docker containers and Helm charts locally. The second part uses the same techniques in the cloud.

Installing & Running the HPCC Systems Platform

These instructions will guide you through installing and running the HPCC Systems Community Edition on a single node to start and then optionally, expand it to a larger cluster of nodes.

Using WsSQL

WsSQL is a service that provides an SQL interface into HPCC Systems. This web service accepts a subset of prepared and standard SQL queries. Learn how to use WsSQL to get access to most HPCC Systems and published queries.

Using Configuration Manager

This document covers configuring an HPCC Systems environment using the Configuration Manager, including Advanced View.

HPCC Systems Administrator’s Guide

These instructions will guide you through installing and running the HPCC Systems Community Edition on a single node to start and then optionally, expand it to a larger cluster of nodes.

Running HPCC Systems in a Virtual Machine

This outlines the system requirements and steps for installing and running an HPCC Systems platform on a single node inside a virtual machine.

Please Note: HPCC Systems 8.2.0 is the last release that will include a Virtual Machine image, which can be used under Oracle VM VirtualBox™ to bring up a single-node system in a virtual machine. More details of how to run the HPCC Systems platform on a local machine (for version 8.4.0, and forward) can be found here.

HPCC Systems Preflight and Certification

This document provides steps you can use to certify your HPCC Systems environment. Use this procedure when you create a new HPCC Systems cluster or after you make any changes or upgrades to the system. This will ensure that your system is functioning properly.

HPCC Systems Monitoring and Reporting (Technical Preview)

This document describes the technical preview of the graphical monitoring and reporting components in the HPCC Systems platform.

ECL Language Reference

ECL is the Enterprise Control Language designed specifically for huge data projects using the HPCC Systems platform. Its extreme scalability comes from a design that allows you to leverage every query you create for re-use in subsequent queries as needed. To do this, ECL takes a dictionary approach to building queries wherein each ECL definition defines an Attribute. Each previously defined Attribute can then be used in succeeding ECL Attribute definitions as the language extends itself as you use it.

Standard Library Reference

The Standard Library Reference includes utility functions for string handling, date manipulation and others.

ECL Programmer’s Guide

The ECL Programmers Guide gives an introduction to the ECL language along with example data and use cases. The supporting ECL Code Files referenced in the guide are included below.

Comrise, a HPCC Systems partner, has also made available a Chinese translated version of the ECL Programmers Guide (version 4.2.0.3rc).

HPCC Systems Data Tutorial: with an Introduction to Thor and ROXIE

This tutorial provides a walk-through of the development process from beginning to end and is designed to be an introduction to working with data on HPCC Systems, as well as give an introduction to Thor and Roxie. The Sample Data Files are required references for the tutorial.

ROXIE: The Rapid Data Delivery Engine

This guide is an introduction to the Rapid Data Delivery Engine (ROXIE).

The ECL IDE and HPCC Systems Client Tools

This manual contains documentation for the set of Client Tools for use with the HPCC Systems platform. These tools include: ECL IDE, ECL Command Line, ECL Compiler, ESDL Command Line, and DFUPlus.

Client Tools

This manual contains documentation for the set of Client Tools for use with the HPCC Systems platform. These tools include: ECL Command Line, ECL Compiler, ESDL Command Line, and DFUPlus.

ECL Watch

This document describes ECL Watch, an interface to the HPCC Systems platform that allows you to view information and interrogate nodes to confirm all expected processes are running. This provides a simple Web-based view into the system and a means to perform Workunit and data files maintenance.

Six Degrees of Kevin Bacon ECL Example

This step by step ECL Programming Example details how to solve the challenge of finding the Six degrees of Separation between any actor and Kevin Bacon.

^{The information used for this analysis can be found on www.imdb.com.}

ECL Best Practices

This guide is a collection of tips and techniques that target the best practices of ECL programming.

View the PDF

ECL Playground

ECL Playground is a tool hosted on an ESP server. A page runs in your browser, allowing you to access and execute self-contained ECL code on your HPCC Systems platform without the use of any other tools. The ECL Playground then shows you the results and the graph in your browser. The view is very similar to what the ECL IDE displays.

Listen to the ECL Playground Podcast and how it can be a helpful tool for developers.

Visualizing ECL Results

This document covers the Visualizer Bundle–an open-source add-on to the HPCC Systems platform to allow you to create visualizations from the results of queries written in ECL.

Visualizations are an important means of conveying information from massive data. A good visual representation can help a human produce actionable analysis. A visually comprehensive representation of the information can help make the obscure more obvious. Pie Charts, Line graphs, Maps, and other visual graphs help us understand the answers found in data queries. Crunching big data is only part of a solution; we must be able to make sense of the data, too. Data visualizations simplify the complex.

The Visualizer bundle extends the HPCC Systems platform’s functionality by allowing you to plot your data onto charts, graphs, and maps to add a visual representation that can be easily understood. In addition, the underlying visualization framework supports advanced features to allow you to combine graphs to make interactive dashboards.

ECL Scheduler

The ECL Scheduler is a component process installed with the HPCC Systems platform. ECL Scheduling provided a means of automating processes within ECL code or to chain processes together to work in sequence.

Data Handling

These steps explain how to move data on or off an HPCC Systems Cluster.

Dynamic ESDL

This manual covers Dynamic ESDL (Enterprise Service Description Language) –a methodology that helps you develop and manage web-based query interfaces quickly and consistently.

Dynamic ESDL takes an interface-first development approach. It leverages the ESDL Language to create a common interface “contract” that both ROXIE Query and Web interface developers will adhere to. It is intended to allow developers to create production web services, with clean interfaces that can evolve and grow over time without breaking existing applications.

ESDL Language Reference

The ESDL Language Reference covers the ESDL (Enterprise Service Description Language) and its syntax. ESDL is a methodology that helps you develop and manage web-based query interfaces quickly and consistently. This goes hand-in-hand with Dynamic ESDL.

Security Manager Plug-in Framework

The Security Manager Plug-in framework provides a mechanism for the creation and deployment of custom security manager plug-in implementations.

Configure an HPCC Systems Cluster on AWS

Running an HPCC Systems cluster inside an Amazon Web Services (AWS) Elastic Cloud (EC2) allows you to instantiate and run HPCC Systems clusters of different sizes on the fly.

This is useful for:

• Proof-of-concept
• Experimentation
• Learning
• Leveraging the HPCC Systems platform without incurring cost of hardware and administration
• Create and use an HPCC Systems cluster immediately without purchasing and installing new hardware

You can create a small cluster for small tasks or larger clusters for larger jobs. This flexibility allows you to match cost and processing power to the job at hand.

Temporary EC2 nodes allow you to “rent” computing capacity without long term commitments. In this manner, you pay as you go instead of incurring large fixed costs at the start. Keep in mind that you should terminate any unneeded instances to avoid paying for computing time you don’t need.

System Requirements

The following system requirements cover what is needed to support the HPCC Systems Platform and ECL IDE included in the Community Edition.

View HTML

Red Book and Community Wiki

The HPCC Systems Community Wiki includes information provided by the community covering best practices, tips, sample code and examples. The HPCC Systems Red Book also contains useful information to help users manage the transition between releases.

View HTML