Tombolo - Zookeeper and Kafka

[rfernandez2007]

Hi,

I have a few questions about Tombolo installation.

In the installation process, particularly in Docker there are Zookeeper and Kafka configuration sections.

1. Are they a mandatory part of the Tombolo architecture?
2. In case they are not mandatory, how do you remove them without breaking the installation?
3. In case they are mandatory, is it possible to configure them without all the security features, like for example certificates, keystores, etc? How would you achieve this?
4. In case it is not possible to avoid installing zookeeper and kafka as part of the stack, and all the security must be configured, could you please provide sample configuration values that help understand the relationship between all the parts, and allow to configure a working installation.

I've previously worked with zookeeper and kafka in other Docker configurations, but the setup was much simpler.

5. Also, there are some variables to be filled that are not clear (to me) to where they belong and what's their function:
JOB_COMPLETE_TOPIC=
START_JOB_TOPIC
JOB_COMPLETE_GROUP_ID=
JOB_COMPLETE_TOPIC=


Any help in configuring/understanding this will be much appreciated!!

Thank you very much!!
Warm regards
Ricardo

[jjacob]

Hello,

Kafka and Zookeeper setup are optional for Tombolo. At some point, Tombolo used Kafka and Zookeeper for Orchestrating workflows, but the architecture has changed and they are not used for Orchestration currently. However, a Kafka instance is required if you want to trigger a job based on Kafka messages.

The following configurations are no longer needed except for START_JOB_TOPIC, which is the Topic Tombolo is listening to trigger Kafka based jobs. However, for basic Orchestration it is not needed

JOB_COMPLETE_TOPIC=
START_JOB_TOPIC
JOB_COMPLETE_GROUP_ID=
JOB_COMPLETE_TOPIC=

Thanks

[rfernandez2007]

Hello Jerry,

Ok, that's good to know, and I'll take it into consideration for design purposes.

Thank you!
Regards!
Ricardo