5 Questions with Anthony Fishbeck

Anthony Fishbeck is a senior architect for LexisNexis Risk Solutions and the creator of the Enterprise Services Platform (ESP), a highly adaptive and extensible component that serves as the front end of HPCC Systems.

Anthony is a known innovator who has worked with large-scale data technologies for more than 18 years. He was also recently recognized by LexisNexis Risk Solutions for his contributions to the organization and the development of HPCC Systems.

Our Vice President of Technology, Flavio Villanustre, sat down with Anthony to discuss his development of ESP, his love for large-scale data, and his process for innovation. 

What were the guiding principles for the development of ESP?

ESP began as a project to create a farm of distributed image servers that would handle a massive number of JPEG images for a project we were developing. At the time, we were working in a dynamic environment – very startup oriented. You never knew what was going to happen next, so I wanted to create a platform that I could continue to develop and grow as new projects came out. 

The platform had to be fast and high-performance, but I also wanted something that was very flexible and adaptable. For example, when we acquired another company and had to integrate our technology with their existing systems comprised of proprietary protocols and back-end systems, we used ESP. It was through ESP’s flexibility and adaptability that we plugged all the HPCC Systems technology into that existing environment without having to drastically change other components.

What makes ESP so adaptive?

One aspect that makes ESP so adaptive is that almost everything in the platform is a pluggable component that is all tied together at configuration time. For instance, if you wanted to integrate with a particular security back-end system and were accessing it through one of our proprietary protocols or a web service, ESP could help harmonize these different components. This is done through the platform’s ability to tie all of these components together with their abstract interfaces, and suddenly, your service is speaking the same web service language or another proprietary protocol. 

As you define a service, you use an interface definition language, which we call the ESDL. Beyond just describing the format of the message, ESDL allows us to programmatically map it to all the different protocols. 

For example, if I’m using that proprietary protocol that I was talking about – accessing the same service as a web service or doing REST with URL-based processing – it automatically matches the language to the service. Additionally, the definition protocol language allows you to version the interface. Things are constantly changing at HPCC Systems, so the same interfaces have to adapt, and the same services need to support new functionality. If you look at how a lot of the web services were developed in the past with the sub standards, it essentially meant that if you made a change, you were publishing a new interface. Our technology enables us to continue to evolve without affecting the clients, just by creating new versions of the existing interface.

What were some of the biggest challenges you faced when designing ESP, and how did you overcome them? 

We have a few different flavors of ESP and the way that it handles dynamic operations. The ESDL definition of the interface generates code that handles all of the processing of the messages. So again, the service writer doesn’t need to understand it, but it’s also generated so that it’s efficient. It’s not walking through the definition of the interface at runtime. That initial binary or static way of creating these interfaces was very efficient and generated before it was built. We moved away from that slightly because you can get stuck in this release cycle. We do have new technologies that allow you to publish the interface on the fly at runtime, which does have a small overhead of processing the interface as it goes, but it’s all pre-processed and turned into a pretty efficient set of internal logics. 

Another challenge we faced early on was dealing with web service standards. If you think back to 2001 when we started creating ESP, SOAP was not yet the de-facto standard for web services. That’s when I started thinking about how to standardize the interfaces of this component. SOAP and XML-RPC were both battling it out for web service standards, so I was implementing web services, making all these components pluggable, while at the same time trying to decide which standard to reference.

Luckily, I got it right and picked SOAP as the initial linemen. At the same time, I was mapping all the HTTP components and that lends itself to the next stage, which of course was REST. Even when we were doing SOAP, we were doing it in a RESTful way because we were mapping out all the HTTP level protocols. 

How have you seen HPCC Systems and its users evolve in the last five years? Where do you see the platform in the next five years?

This is a hard question because there’s been so many developments over the years. One example is how much support has been added towards integrating HPCC Systems with other technologies like Apache Spark. The HPCC Systems team never seems to rest on their laurels. When they start integrating a technology, they learn from the process. For example, the work done providing file access to Spark was an opportunity for HPCC Systems to support remote filtering of data at the remote file server. This doesn’t just help with external technologies, but it also enables us to further optimize how the same work is done for ECL queries.

As for the next five years, there’s going to be a big effort toward moving our technology into the cloud, but again, the HPCC Systems team won’t just be doing a lift and shift of the technology. This will be an opportunity to enhance how we dynamically manage our platform and clusters; like instantiating clusters of various sizes on the fly as well as how we best make use of shared resources while we work to make full and efficient use of cloud storage mechanisms and other resources.

Earlier this year you were recognized by LexisNexis Risk Solutions for your innovative contributions to HPCC Systems development. What do you think is the key to excellent innovation, and do you have any tips to share? 

One of the keys is instead of looking for a new solution, I’m looking for new ways of thinking about the problem. if somebody comes to me asking for a particular solution, I ask them, “What is the problem you’re trying to solve?” because I want to understand it and look at it from different angles, which can help me recognize patterns. It involves taking a step back, ignoring the solution that people are proposing, and thinking in multiple ways about the problem before you start to take action.

Want to hear Anthony and Flavio’s full conversation?  Listen to the webcast.

(Note: Not all questions were recorded due to technical difficulties but are included in this write up.)