The Download: Tech Talks by the HPCC Systems Community, Episode 9
On November 16, 2017, HPCC Systems hosted the latest edition of The Download: Tech Talks. These technically-focused talks are for the community, by the community. The Download: Tech Talks is intended to provide continuing education through high quality content and meaningful development insight throughout the year.
Links to resources mentioned in Tech Talk 9:
Episode Guest Speakers and Subjects:
Robert Pelley, Architect for UK and Ireland, LexisNexis Risk
Robert has over 30 years of experience in software and systems engineering and has been with LexisNexis for 6 years. In prior roles, Rob worked extensively with relational databases and having designed and built a multi-terabyte data warehouse, he began to find the limits of relational technology with big data. Rob started with LexisNexis in 2011 and has been working extensively with HPCC Systems based solutions. Rob has special interest in software and systems engineering for big data in very high volume transaction processing environments.
Bob Foreman, Senior Software Engineer, LexisNexis Risk Solutions
Bob Foreman has worked with the HPCC Systems technology platform and the ECL programming language for over 5 years, and has been a technical trainer for over 25 years. He is the developer and designer of the HPCC Systems Online Training Courses, and is the Senior Instructor for all classroom and Webex/Lync based training.
Key Discussion Topics:
1:22- Flavio Villanustre provides community updates:
HPCC Systems Platform updates
- 6.4.4 gold is now available
2018 Summer Internship Proposal Period Now Open
- Interested candidates can submit proposals from the Ideas List
- Visit the Student Wiki for more details
- Deadline to submit is April 6, 2018
- Program runs late May through mid-August
2017 HPCC Systems Community Day recordings available
- Thank you to our Sponsors and Speakers for making this year a success!
- To Sum it up: 4 Tracks, 16 Talks, 24 Speakers, 180 Attendees, 10 Posters, and 2 Autonomous Robots!
- Watch at https://hpccsystems.com/hpccsummit2017
- Read a recap on the Poster Competition
- Details on 2018 HPCC Systems Community Day to be announced in early Q1. Stay tuned!
7:18 – Robert Pelley, Architect for UK and Ireland, LexisNexis Risk Solutions – Integrating REDIS with HPCC Systems in high volume UK infrastructure.
REDIS (REmote DIctionary Server), an in-memory caching technology will be integrated into the LexisNexis application stack to provide an additional cache at the entry point of the application infrastructure. This will serve as a cache of Product Responses whereas existing caches will continue to serve Vendor Responses. The aim of the REDIS front-end Product Response cache is to improve system throughput and response times.
Robert will review the impact of rapid growth in Transactional Volume and then walk us through analyzing and understanding request duplication. He then takes a look at UK HPCC Architecture before and after REDIS Caching and talks about the benefits of REDIS Caching. He ends the presentations with some interesting final thoughts and Q&A.
21:20 – Q&A
Q: How does REDIS manage memory, so that the cache does not grow uncontrollably?
A: Every time you set a key in the cache, you have the option to set an expiry. Ours was a 15 minute cache, so we set every key value with a 15 minute expiry, which means that after 15 minutes, it disappears from the cache. Then you have to reset the cache subsequently. Essentially, each cache entry expires after a specific length of time.
Q: What hashtag function is used in ECL to derive the hash key?
A: Good question! In our implementation where we genericized the context and request and added those strings together and derived a hash key, we used the hash MD5 function to provide a hash key. Hash MD5 is the ECL function that is readily available.
Q: How is REDIS called in ECL via embed or direct function call?
A: REDIS is an ECL Plugin and so EMBED is not required. So use IMPORT redisServer FROM lib_redis:
If you have additional questions, please contact Robert Pelley.
24:00 – Bob Foreman, Senior Software Engineer, LexisNexis Risk Solutions – ECL Tips: The Bright Green Data Generation Machine (DataGen)
Bob will be talking about the ECL Code Generator, DataGen. He starts by reviewing the evolution of DataGen and then talks about its requirements, installation requirements and how to actually use it. He wraps things up with a live demo to show DataGen in action!
If you have additional questions, please contact Bob Foreman.
Have a new success story to share? We would welcome you to be a speaker at one of our upcoming The Download: Tech Talks episodes.
- Want to pitch a new use case?
- Have a new HPCC Systems application you want to demo?
- Want to share some helpful ECL tips and sample code?
- Have a new suggestion for the roadmap?
Be a featured speaker for an upcoming episode! Email your idea to Techtalks@hpccsystems.com
Visit The Download Tech Talks wiki for more information about previous speakers and topics: https://hpccsystems.atlassian.net/wiki/display/hpcc/HPCC+Systems+Tech+Talks
Watch past episodes of The Download
Upcoming Events:
Our next episode of Tech Talks will take place on January 18. Click here for more details and registration!