Home > Deploy > Third Party Integrations > Spark-HPCC Systems Integration

Spark-HPCC Systems Integration

HPCC Systems-Spark support consists of a Java library that facilitates access from a Spark cluster to/and from data stored on an HPCC Systems cluster.

HPCC Systems-Spark Support

The Spark-HPCC Systems Distributed Spark Connector employs the standard remote file read facility to read and write data to/from either sequential or indexed HPCC datasets.

The data on an HPCC cluster is partitioned horizontally, with data on each cluster node. Once configured, the HPCC data is available for reading and writing in parallel by the Spark cluster.

The HPCC Systems Spark Connector requires Spark 2.4.6, 3.4.1, or 3.2.3 and the org.hpccsystems.wsclient library available from the Maven Repository.

  • Find the source code and examples in the spark-hpccsystems repository
  • Get the latest JAR or javadocs files from the Maven Repository
  • Example Maven dependency information, be sure to update the <version> with the appropriate version you are using:
<dependency>

      <groupId>org.hpccsystems</groupid>

      <artifactid>spark-hpcc</artifactid>

      <version>9.4.0</version>

</dependency>

 Note: This project references log4j which has been reported to include security vulnerabilitie(s) in versions prior to v2.15.0
The Spark-HPCC project no longer references the offending log4j versions. Users of Spark-HPCC are strongly encouraged to update to the latest version.
Learn more about the vulnerabiltiy: https://github.com/advisories/GHSA-jfh8-c2jp-5v3q

Please Note: The HPCC integrated Spark plugin is no longer supported as of version 9.0.0 in favor of stand-alone user-managed Spark clusters linked to the HPCC platform using the Spark-HPCC connector.