When is an ESP not an ESP?

I’m sure we have all got in to one of those conversations where we think we’re talking about the same thing as another person, but down the line realise we’re talking at cross purposes. At times, having a conversation about ESP can be just like this. ‘What do you mean?’ I hear you say. ‘I know exactly what an ESP is.’ But do you really?

ESP (Enterprise Services Platform) is the container which allows multiple services to be ‘plugged in’ which provide various types of functionality to client applications. For example, WsECL, which allows you to interface to published queries on Roxie or Thor, and ECL Watch which is our web-based query execution, monitoring and file management interface. ESP supports a number of protocols including, SOAP, XML, HTTP, REST and JSON. 

Unknown Object

ESP – An adaptive container for hosting web services

We often talk about running an ESP, how we use an ESP, or what functionality it does or doesn’t provide for us.  But ESP is really more of a container with lots of attachments that allow it to speak a particular protocol, such as HTTP, integrate with a particular security system, or perform transaction logging, etc.  But what is in that container are the services that provide all the custom logic that performs the tasks we want. These services can have extremely different characteristics from one another, they can: 

  • Have extremely different performance characteristics
  • Make extremely different trade-offs
  • Perform extremely different tasks. 

When discussing these services it’s almost always better to name the service itself rather than calling it an ESP.  Naming the service immediately brings to mind its specific functionality, trade-offs, and design.

When web developers write a javascript function that make a REST call to WsFileSpray in order to de-spray a file from THOR, they’re calling out to an ESP.  When an ECL developer makes a SOAPCALL out to WsPackageProcess to programmatically update their packagemap they’re calling out to an ESP.  When you run ‘ecl publish’ from the command line, which makes calls out to WsWorkunits to publish your query, it’s calling out to an ESP.

Perhaps you are tempted to assume that WsFilespray, WsPackageProcess, WsWorkunits and other Ws<Type> functions are actually an ESP in their own right? If you are, then I’m afraid you’re wrong. They may call out to an ESP but they aren’t ESPs themselves, they are Web Services hosted by an ESP and they come in many forms.

HPCC Systems® Operational Web Services

If you are using WsWorkunits, WsDfu, WsFileSpray, WsPackageProcess, WsTopology or any other components that provide HPCC Systems® related functionality, then you are using HPCC Systems® Operational Web Services.

Product Web Services

If you have a user defined service such as Ws_MyProductName, which might be used to provide services to one of our applications, you are using a Product Web Service.  Product web services have interfaces defined using the ESDL interface definition language, but those interfaces are static, compiled and “built in”.

dynamicESDL

If you are dynamically publishing ESDL interfaces in front of ESDL based roxie queries, you are using dynamicESDL.  It provides interface management, versioning, security and transaction logging without having to do any coding beyond the definition of the query interface using the ESDL interface definition language.

ECLWatch

If you’re using HPCC Systems®, then the chances are you are using ECL Watch. You’re monitoring the health of the various components and clusters, keeping an eye on the job queue and deployed roxie queries, spraying and despraying files to and from the landing zone etc.

Is this ESP? It’s no again, I’m afraid.

While your ECL Watch is accessed via a port on the ESP (8010) it isn’t the ESP itself.  Once again what you’re really interacting with are those services hosted within the ESP container only this time there is a layer of javascript that mashes those services all together and provides the user interface that we normally think of when we talk about ECLWatch.

WsECL – Is it an ESP, ESP Service or ROXIE?

WsECL can get even more confusing.  Many people think that the web page WsECL, is actually Roxie, probably because it allows you to interact with your published Roxie queries.  As with other services, it is also often wrongly referred to as ‘the ESP’. 

WsECL is a little bit special in how it functions inside the ESP.  Like all services it sits inside ESP’s service container.  And like all services it has its own functionality and performance characteristics.  But what makes WsECL unique is how this service takes published ECL queries and makes them look like services of their own.  WsECL is sort of a service of services.

WsECL provides a simple way of interacting with ECL code that is parameterized and can be run time and time again.  The query could be doing anything, although it only really makes sense to use WsECL for repeatable processes and queries that differ based on which values are input. 

When you publish a query and access it via WsECL, a form is provided making it easy to interact with your query, by allowing you to set values and submit it for a run.  A wsdl is provided to make it easy to integrate other applications that may want to call your query.  Sample requests and responses are provided.  Results can be shown in different formats, rendered as a table, or custom rendered based on XSLTs that were published with the query.

Just to be clear WsECL isn’t Roxie or ESP. Again, it’s another service that sits inside the ESP ‘container’.

What is ESP?

ESP (Enterprise Services Platform) is the host for all of these web services functions. It’s a container for running services.  ESP is what provides you with the ability to write and run web services, support revenue based products and use the dynamic ESDL engine which gives you flexibility and ease of use.  ESP may also be referred to as the host for the collection of adaptive components that lets your services interact with the outside world using a protocol like REST, or SOAP, or LDAP security for example. 

These bits of functionality are really the only things that should ever be referred to as ‘the ESP’. Anything else, is a service. Take a look at this diagram which illustrates the different relationships the various services have with the ESP:

Troubleshooting ESP

Most problems are likely to be related to the individual service you are trying to call, or the user interface layer of javascript that makes up an interface like ECLWatch. 

Issues that affect multiple services are more likely to be related to a shared component that lets ESP adapt to those systems or protocols, especially if it affects:

  • How client applications communicate with the service
  • How it interacts with the security systems on the back end
  • Transaction logging

Problems like these might well be talked about as ESP issues.  Although, it’s worth noting that components that just happen to be shared by a few closely related services and aren’t directly accessed by the ESP container, should be considered part of those shared services.

If you’re not sure whether a problem is with the service you are calling or with components that are closer to the core of ESP, there’s really only one way to find out, ask an HPCC Systems® platform developer.

Hopefully, this helps to cast aside some common misconceptions about ESP and the services it hosts, while clarifying why the root of a problem you may be experiencing, may not be with ESP at all, but with the service you are using. 

Notes:

  1. Watch a video about ESP and Client Tools.
  2. Find out more about ECL Watch.
  3. Find out more about Dynamic ESDL.
  4. Find out more about how HPCC Systems works.  
  5. Take a tutorial to learn about how Thor and Roxie work.
  6. If you want to discuss an issue about ESP or any other component with an HPCC Systems® platform developer, use our Community Issue Tracker. For automatic updates, create a user account. You can also post questions in the Developer Forum.