Thread by @StephanEwen

Today, we announced @dataArtisans Streaming Ledger: Serializable ACID transactions across multiple tables, keys, and streams, in pure stream processing. We are removing the long standing limitation that streaming applications can only operate consistently on one key at a time.

Many applications need to guarantee integrity of data, with constraints across many tables, rows, entries, or keys. Examples are plentiful anywhere you deal with financial data, in master data management, in logistics, etc.
All these applications now become streamable.

I believe that this technology is the most significant addition to stream processing guarantees since exactly-once semantics were introduced a few years back. Exactly-once stateful stream processing is a commodity now, this is the new frontier of development.

Streaming ACID transactions complement the technology landscape: Like stateful stream processing combines event-driven computation and key/value-store-like state, streaming ledger combines streaming computation with relational-database-style consistent overall view of the data.

Rather than building on traditional optimistic/pessimistic transaction processing approaches, we came up with a new technology that is crafted to exploit the advantages of a stream processing: We exploit logical event-time, out-of-order-processing, and iterative dataflows.

The result is a processing scheme that guarantees serializable isolation by scheduling all state accesses and updates such that they provably obey a conflict-free schedule. The guarantees are better than in some of the world's best selling relational databases.

The approach also seems to be really fast. Without any specific work in performance optimization so far, we tested this at 2 million txns/sec across two tables of 100 million rows using 32 slots in Flink. 100% txn update rate, each txn mutating four rows each. Good start...

Find out more, like concepts, APIs, example on GitHub:
github.com/dataArtisans/d…

Please contact us if you want to take this for a test-drive:
data-artisans.com/streaming-ledg…