Thread by @gunnarmorling

Message transformations (SMTs) are an invaluable feature of @ApacheKafka Connect, enabling tons of use cases with a small bit of coding, or even just configuration of existing SMTs ready to use. Here are some applications in the context of change data capture: (1/7)

* Converting data types and formats: date/time formats are the most common example here, e.g. to convert milli-seconds timestamps into strings adhering to a specific date format (2/7)

* Creating an "anti-corruption layer", shielding consumers from legacy schemas or ensuring compatibility after schema changes; e.g. could use an SMT to choose more meaningful field names, or re-add a field using its old name after a column rename, easing consumer migration (3/7)

* Removing fields or extracting parts from messages: sometimes you'd like to exclude some fields when writing to a specific sink, or you'd like to extract just the new row state of CDC messages with both old and new row state (4/7)

* Routing messages to a different topic; e.g. with #Debezium you'll have a 1:1 relationship between captured tables and corresponding topics. Thinking of sharding use cases, you might want to stream changes from multiple tables to a single topic. No problem with an SMT. (5/7)

Debezium comes with a set of SMTs for some of these purposes:
* Extracting new row state: debezium.io/docs/configura…
* Topic routing: debezium.io/docs/configura…
* Routing messages from an outbox table: debezium.io/docs/configura…

(6/7)

Also, every so often when asked "How to do XYZ with Debezium?" and it's too specific of a requirement for supporting it upstream, the answer is: "Just implement a custom SMT". SMTs are a powerful tool in the box that every user of Kafka Connect should be aware of!

~Fin~