Thread by @MatthiasJSax

Insight of the day: #ApacheKafka actually implements a three-phase commit protocol (not just the standard two-phase protocol) for its transactional writes . \1

We usually explain it as a two-phase commit, and the third phase is somewhat hidden. The reason is, that the coordination of the protocol is split between the producer and the broker TX-coordinator.

The producer waits until all writes are flushed, replicated, and acked before it starts the commit. This waiting completes the first phase. \3

The actual commitTx call from the producer to the broker TX-coordinator completes the second phase. (We often describe this as the end of the first phase of the commit, as the data is not readable yet, because no commit markers are written to the data partition yet). \4

However the waiting of the producer effectively ensures, that the broker side TX-coordinator conceptually transit into preparedCommit state! I always missed this detail! Read up at en.wikipedia.org/wiki/Three-pha… \5

Why does it matter? Well, the standard two-phase commit protocol could block for some error cases, while the three-phase commit protocol helps to avoid blocking for those cases! \6

Thanks to @hostirosti who asked about it! After he pointed out the difference between both protocols I read up on it and discover this sweet little detail! Always good to discuss with @hostirost!

\theEnd