The Download: Tech Talks by the HPCC Systems® Community, Episode 13
On April 19, 2018, HPCC Systems hosted the latest edition of The Download: Tech Talks. This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation, and further build and link the relationships within our HPCC Systems community.
Links to resources mentioned in Tech Talk 13:
HTPASSWD Reference Model Source Code
Episode Guest Speakers and Subjects
Matthew Upchurch-Butler, HPCC Systems® Intern, Kennesaw State University – The Future of Automotive Telemetry: Assessing Autonomous Vehicle Risk Implications using Simulated Data
Matt is currently a senior at Kennesaw State University majoring in Information Technology with a concentration in Information Security and Assurance. Last year Kennesaw State held its first annual Hackathon. The Hackathon featured outside companies with real world problems for the students to work through. Matt partnered with the HPCC Systems team and was named the third-place winner. He joined the LexisNexis family as an intern in February of 2018. Presently, Matt is working to develop a standard Math library for the company’s Enterprise Control Language (ECL).
Russ Whitehead Architect LexisNexis Risk Solutions – Developing A Custom, Pluggable HPCC Systems Security Manager
Russ Whitehead (William) is a Software Architect on the HPCC Systems Platform and has a BS degree in Computer Science from University of Florida, and an MBA from the University of Miami. His background is operating systems development, network management and voice recognition, and he has been a LexisNexis employee and a platform team member since May 2008. His top responsibility is the HPCC Systems security framework. Russ also serves as a member on the HPCC Systems Community Day Review Committee.
Craig Cooper Consulting Software Engineer LexisNexis® Risk Solutions – Workunit Graphing Feature of the ECL Watch
Craig is a 26-year employee of LexisNexis. With 20+ years of object-oriented programming experience in the Windows environment Craig has been transitioning to HPCC Systems and ECL over the past year. Craig also has more than 15 years of experience (part time) teaching software concepts at the college level.
Key Discussion Topics:
1:19– Jessica Lorti provides community updates:
HPCC Systems Platform updates
• 6.4.14 is the latest gold version
• View the changelog for details
Summer Internship Program
• Excited to accept 10 interns into the program this year!
• Details on the students and projects coming in May
Recent Blogs
• Fly on the wall – Developer offsite meeting
• Software Architecture Design and How to Implement It in ECL
• HPCC Systems Commits to Fund the Planting of 10K Trees
2018 HPCC Systems Summit Community Day
• October 8-9 in Atlanta
• Two pre-event workshops this year
• Poster Competition
• Sponsorship packages
• Now Open – Call for Presentations and Poster Abstracts
• View more at hpccsystems.com/hpccsummit2018
10K Trees Campaign for Earth Day
World Planting Day, March 21 through Earth Day on April 22
• Help us help the environment on behalf of our community!
• HPCC Systems is dedicated to the environment and is giving you the opportunity to take action and be a small part of a big impact.
• HPCC Systems, partnering with the National Forest Foundation, is growing and promoting awareness of environmental sustainability with their 10,000 Trees challenge.
8:20- Matthew Upchurch-Butler, HPCCSystems® Intern, Kennesaw State University – The Future of Automotive Telemetry: Assessing Autonomous Vehicle Risk Implications using Simulated Data
In his presentation, Matt gives us an interesting look into the concept of self-driving vehicles. Something that has fascinated mankind for generations. Once thought to be a topic so farfetched that it could only exist in a child’s cartoon, today we seem to be living in that reality. Several companies have taken major strides toward making this once fanciful idea a reality. But as we race closer and closer to that dream, we stop and pause to consider the risks and challenges that we may be creating for ourselves. Beyond that, how will this new technology affect today’s modern societal and industrial norm? It is an eye opening and relevant topic that everyone will enjoy!
31:48- Q&A
Q: How do you think using autonomous vehicles will impact humane behavior/society?
A: I think they will make us faster and it will be easier to get around, but there is also a potential risk that it will make us lazy as a society. As humans, we tend to rely on automated systems and we trust that things will get done. Skills that we don’t use or skills that we allow computers to do for us can result in our skills getting rusty. So, we might forget how to drive. Like everything, we need to proceed with caution and we should not go overboard with it.
Q: How dangerous do you think self-driving vehicles are compared to human drivers?
A: Based on the research I have done, statistically speaking, there is less risk in autonomous vehicles. The numbers show that and the math proves that. As humans, if we give a machine the authority to regulate a process, then we need to make sure that process is executed 100% of the time. The numbers are there and it is less dangerous, but how much risk are we willing to accept as people from a computerized process? That is something you need to consider on your own as a person.
Q: Legally speaking, how has the government responded to use and growth of the autonomous vehicle segment?
A: There are a couple of different ways to look at that question. In the presentation, I mentioned 21 states have passed or introduced legislation for self-driving vehicles, but 29 states have no regulations. States like Arizona, where the woman was killed, have very loose regulations and these are the states that companies are selecting to do their testing in because of the lightened legal restrictions there. Federally, we have not done very much, but I think that is coming. Overall, laws are made state by state and some states factor in the protection of others and some do not.
Q: Is the work that you are doing with the math library similar to any work you have done in the past? And we are thrilled to have you as an intern here working on this project.
A: Thank you. I am thrilled to be here. Is the math library similar to other work I have done? A little. I have studied object-oriented programming for a while now. My major now is IT, but previously I studied computer engineering and electrical engineering, so I do have a strong background with programming. It is similar because object-oriented concepts do apply across different languages. Different syntax, but they can feel familiar. Beyond that, not really. This has been a new learning experience for me. I feel confident about it, but it is a little similar and a little new and fresh.
Q: Would you say future insurance premium calculations will be based on the age of the car, manufacturer, etc. as opposed to the age of the driver as is currently done?
A: I think it will be both. I think they will factor in the age of the vehicle, the manufacturer, but they can get more specific once these vehicles control themselves. For example, what version of the software do you have? Have we had issues with the security flaws in previous versions? Did you get this patched? Maintenance records could also be better tracked (like oil changes). More information can be calculated based on the vehicle itself and also the driver. We are moving toward individual based behavioral calculations. So, who typed in their password or scanned their thumb print and then took riskier routes? We are going to have very specific insurance-based calculations in the future.
If you have additional questions, please contact Matthew Upchurch-Butler.
40:30 – Russ Whitehead Architect LexisNexis Risk Solutions – Developing A Custom, Pluggable HPCC Systems Security Manager
HPCC Systems currently provides several out of the box security managers, which range from simple HTPASSWD user authentication to robust LDAP authentication and feature level authorization. In his presentation, Russ will describe how a software developer can implement the HPCC Systems security manager interface in order to create a custom security manager that can be plugged into the HPCC Systems configuration. With this framework, a developer can implement a security manager that supports fingerprints, voice recognition, retinal scans or any other mechanism they choose.
50:59 – Q&A
Q: Can ESP be configured to use one security manager and Dali configured to use another?
A: Absolutely. This is one of the things we planned for as we designed it. Not only can ESP use one and Dali use another – those are just two HPCC components. As I showed in the example, ESP can be configured to use different ones. Like I said, if you have different security requirements for the different services within ESP, then absolutely you can.
Q: Can a custom pluggable security manager be used by components other than ESP such as Dali?
A: Yes, they all can.
Q: Can you restrict access to the landing zone folders working as a file scope using the security manager?
A: Yes, you can. That is one of our advanced topics that our LDAP security manager does. The HTPASSWD user authentication one only authenticates you. Once you are authenticated, you can do everything. With LDAP or any more robust security manager you want to implement, then you create what’s called features. Give a feature a name like landing zone or file spray. Then you can assign users or groups permission to read it, delete it and so on. Through the ISecManager interface you can do all of these things.
Q: How does an administrator configure feature level permission when using their custom pluggable security manager?
A: If your custom security manger does support feature level authorization then the administrator uses the same interface as they do currently. That means going to security tabs of ECL Watch and there you will find sections where you can create groups, add users to groups, assign various permissions to those groups that restrict or allow access to resources.
If you have additional questions, please contact Russ Whitehead.
1:04:19- Craig Cooper Consulting Software Engineer LexisNexis® Risk Solutions – Workunit Graphing Feature of the ECL Watch
In this edition of ECL Tips, Craig will familiarize HPCC Systems users with the features and uses of the Workunit Graphing functionality built in to the ECL Watch application.
1:29:00 – Q&A
Q: What do you commonly look for in the graph?
A: That is a good question. The first thing I look for are the edges or the record count. In my jobs, I base a lot of expectations on the record counts that I have. If I see a job that runs really long, I am also watching very carefully for skew. Another thing to look for is how many graphs did my job get broken into or did it get broken into additional jobs as well. Is it something that is not getting the amount of parallelism that I would rather prefer. If I have a single graph I will not get a lot of parallel activity for various jobs, so I also want to look at the ECL that I have to see if various segments of it can be forced into parallel or if I can break graphs, so that type of thing might happen.
Q: Is there a legend to describe what each of the icons are in the graph?
A: No, I don’t. What you should use is the label on the activity itself. The label is very descriptive and the icon is there just as a visual aid, but the label on the activity is what you want to look at.
Q: How does skew affect my Workunit?
A: Skew can affect your Workunit in a couple of ways. If you have a heavy skew, that means fewer nodes are participating and your job is going to take longer. Skew can also affect you when you have a heavy skew and the cluster aborts your Workunit. So, you want to look at that and evaluate why you have that skew. It is usually your data. Take a look at the distribution and why you have just one or two nodes participating in the quarry. Distributing the data is critical.
Q: Does one browser display the graphs better than another? What about the version of the browser? Are there any plug- ins required to view the graphs?
A: None of them need plug-ins.
Q: Why are there multiple graphs for just one Workunit?
A: The complier takes a look at the job and sees if it can do any optimization itself by breaking the job into several sub units. These sub units become the graphs. It is the complier that makes the decision.
Q: How are the graphs different for a ROXIE quarry vs a Thor job?
A: I am sorry, but I cannot answer that. My expertise is more on the Thor side.
If you have additional questions, please contact Craig Cooper.
Have a new success story to share? We would welcome you to be a speaker at one of our upcoming The Download: Tech Talks episodes.
- Want to pitch a new use case?
- Have a new HPCC Systems application you want to demo?
- Want to share some helpful ECL tips and sample code?
- Have a new suggestion for the roadmap?
Be a featured speaker for an upcoming episode! Email your idea to Techtalks@hpccsystems.com
Visit The Download Tech Talks wiki for more information about previous speakers and topics: https://hpccsystems.atlassian.net/wiki/display/hpcc/HPCC+Systems+Tech+Talks
Watch past episodes of The Download:
The Download: Tech Talks by the HPCC Systems Community, Episode 4
The Download: Tech Talks by the HPCC Systems Community, Episode 5
The Download: Tech Talks by the HPCC Systems Community, Episode 6
The Download: Tech Talks by the HPCC Systems Community, Episode 7
The Download: Tech Talks by the HPCC Systems Community, Episode 9
The Download: Tech Talk by the HPCC Systems Community, Episode 10
The Download: Tech Talk by the HPCC Systems Community, Episode 11
The Download: Tech Talk by the HPCC Systems Community, Episode 12