Thread by @gunnarmorling

A short 🧵 on @apachekafka topic creation (triggered by @niko_nava, thanks!): who should create Kafka topics, how to make sure they have the right settings, how to avoid dependencies between producer and consumer(s)? Here's my take:

2⃣ Don't use broker-side topic auto-creation! You'll lack fine-grained control over different settings for different topics; Merely polling, or requesting metadata, will trigger creation based on global settings. Plus, some cloud services don't expose auto-creation to begin with.

3⃣ Instead, the producer side should be in charge of creating topics. There you have the information and knowledge about the required settings (replication factor, no. of partitions, retention policy, etc.) for each topic. Depending on your requirements, different approaches...

4⃣ ...work best. If topics are written to by a single producer, roll-out of the producer might also set up / configure the corresponding topics, akin to DB migrations. If you have multiple producers writing to a single topic, or just need more governance in general,...

5⃣ ...upfront coordination via some more formalized process, and topic management through a centralized service, may make sense (depending on your org size, that's a good idea anyways, so to keep track of topic owners, configuration history, etc.).

6⃣ Side-note: auto-creation by Kafka Connect source connectors is an interesting option. KIP-158, added in Kafka 2.6, allows for individual settings (replication, partitions, retention etc.) based on topic name patterns. Details in this post by @rk3rn3r: debezium.io/blog/2020/09/1…

7⃣ How to avoid issues if consumers get rolled out before a topic exists? Don't rely on start-up order -- resilience is key here: e.g. have health checks for Kafka Streams apps to get them automatically restarted. If needed, consumers embedded into a web app may use the...

8⃣ ...admin client API to check the existence of topics, and take some action if they are (or are not yet) available.

Any other tips to add? How do you manage the creation of your Kafka topics? Any best practices to share? Would love to learn from your experiences 🙏.

~~Fin ~~