Mon Jul 04, 2022 5:14 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl (https://stackoverflow.com/search?tab=newest&q=hpcc-ecl). This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.



Tombolo - Zookeeper and Kafka

Post questions specific to installation or configuration for the HPCC Systems platform

Sat Feb 12, 2022 3:14 pm Change Time Zone

Hi,

I have a few questions about Tombolo installation.

In the installation process, particularly in Docker there are Zookeeper and Kafka configuration sections.

1. Are they a mandatory part of the Tombolo architecture?
2. In case they are not mandatory, how do you remove them without breaking the installation?
3. In case they are mandatory, is it possible to configure them without all the security features, like for example certificates, keystores, etc? How would you achieve this?
4. In case it is not possible to avoid installing zookeeper and kafka as part of the stack, and all the security must be configured, could you please provide sample configuration values that help understand the relationship between all the parts, and allow to configure a working installation.

I've previously worked with zookeeper and kafka in other Docker configurations, but the setup was much simpler.

5. Also, there are some variables to be filled that are not clear (to me) to where they belong and what's their function:
JOB_COMPLETE_TOPIC=
START_JOB_TOPIC
JOB_COMPLETE_GROUP_ID=
JOB_COMPLETE_TOPIC=


Any help in configuring/understanding this will be much appreciated!!

Thank you very much!!
Warm regards
Ricardo
rfernandez2007
 
Posts: 25
Joined: Mon Nov 29, 2021 2:40 pm

Tue Feb 15, 2022 2:33 pm Change Time Zone

Hello,

Kafka and Zookeeper setup are optional for Tombolo. At some point, Tombolo used Kafka and Zookeeper for Orchestrating workflows, but the architecture has changed and they are not used for Orchestration currently. However, a Kafka instance is required if you want to trigger a job based on Kafka messages.

The following configurations are no longer needed except for START_JOB_TOPIC, which is the Topic Tombolo is listening to trigger Kafka based jobs. However, for basic Orchestration it is not needed

JOB_COMPLETE_TOPIC=
START_JOB_TOPIC
JOB_COMPLETE_GROUP_ID=
JOB_COMPLETE_TOPIC=

Thanks
jjacob
 
Posts: 7
Joined: Fri Mar 21, 2014 7:14 pm

Tue Feb 15, 2022 4:24 pm Change Time Zone

Hello Jerry,

Ok, that's good to know, and I'll take it into consideration for design purposes.

Thank you!
Regards!
Ricardo
rfernandez2007
 
Posts: 25
Joined: Mon Nov 29, 2021 2:40 pm


Return to Installation

Who is online

Users browsing this forum: No registered users and 1 guest

cron