ECL Syntax Highlighting in GitLab Inc

In recent versions of HPCC Systems, the platform team has been adding colorization features to make it easier for users to read and review ECL code. Colorization is used to help users recognise reserved function names, constants, types, comments and strings in ECL code as well as to distinguish between different types of files and icons when using different editors, such as our own ECL IDE or ECL Extension for VS Code. The list of known colorizers and editors is available in our GitHub repository.

Colorization provides many benefits, not only by making it easier to create and read your own code, but it also helps enormously when reviewing code written by others. It speeds up the process making code development faster and much more efficient. 

While the platform side of this work was ready, there was still work to be done to secure adoption on GitLab so ECL users would be able to reap the benefits of these features.

Adoption on GitLab was finally achieved a few weeks ago and our LexisNexis Risk Solutions Group colleague, Dan Abittan, Senior Software Engineer, has been working behind the scenes to make syntax colouring available to ECL users.

Dan has been working for LexisNexis Risk Solutions Group in the Government business vertical for 6 years.

His role is to develop, innovate and maintain ECL queries, automated macros, web and desktop applications, which impact both our customers and software engineers across a variety of platforms and countries. 

In this blog, Dan walks us through this journey, how it helps ECL users now and future plans for extending the availability of this feature.

The Journey and the Results

A few years ago, our team ported all of our ECL code repositories to GitLab in order to take advantage of the Git workflow.The migration went smoothly, for the most part, and we now had access to a plethora of useful Git features such as Forking, Branching, Pulling and Merging code.

After going through the learning curve, hundreds of ECL developers were now analyzing ECL code on GitLab. However, the ECL code did not look as attractive and something was missing. GitLab did not support ECL syntax highlighting, so we were looking at large ECL files of uncolored text, which looked like this:

Although this may not sound like such a major concern, since most of our users frequently conduct code reviews and analysis on GitLab, investigating large ECL files with no syntax highlighting took a lot more effort and eye strain.

When there is no difference in color between string literals, reserved words and comments, the code just looks disorganized and untidy. This also increases the chances of having both syntax and semantic errors slipping through the cracks and making their way in to QA environments or potentially, even further down the production pipeline. We had to come up with a way to add some color to our ECL code on GitLab. This was easier said than done.

GitLab applies syntax highlighting to native programming languages through Rouge rubygem, which initially did not support ECL. We therefore had to develop a Ruby ECL Lexer and initiate a pull request to Rouge on GitHub. This took a few months of work and consultation with the open source Rouge team.

When our pull request was finally accepted and merged upstream, ECL syntax highlighting became part of the open source Rouge code highlighter. The next stage was to wait for GitLab Inc. to upgrade their Rouge version which they do every few releases. 

The whole process took a total of about 8 months, involving some coding and a little patience for the upgrades to become available. Once the GitLab upgrade was applied to v12.10.10, our ECL code was finally color coded on GitLab. The difference in how the same code now looks visually in comparison with before is clear: 

This useful feature has been very well received. It has instantly benefited hundreds of our ECL users who can now efficiently parse and compare color coded ECL code on GitLab.

Additionally, any open source user running GitLab version 12.10+ now has their ECL syntax highlighted.

Next Steps

Future plans involve getting ECL Syntax highlighting implemented on the GitLab Web IDE which uses yet another format with another flavor of regex lexers (click here for a larger view):

This will be accomplished by updating the Monaco and Ace libraries to support ECL.