,
36 tweets,
4 min read
Read on Twitter
1/ People often ask why it took so long for Kafka to go 1.0.
2/ The answer isn't about stability: Kafka's been in production at thousands of companies for years.
3/ It was about completeness of the vision we were building towards. We didn't want to call it 1.0 when it was only half-way there.
4/ We were a bit pigheaded about this even when big companies told us they couldn't use software that wasn't at least 1.0.
5/ What do I mean by completeness? We built Kafka because we thought there should be an infrastructure platform for streams of data.
6/ For a single application this means scalable event-driven apps and pipelines.
7/ This can grow into a central nervous system connecting services, DBs, data centers, SaaS...everything in the company around streams.
8/ That sounds grandiose, maybe, but there are actually a lot of companies built that way now, using Kafka, which is super cool.
9/ So we wanted to build a streaming platform, not just another message queue. A real, modern distributed infra platform for event streams.
10/ I believe in this even more now--I think the streaming platform is going to be as big and central of a data platform as RDBMSs are.
11/ I think if you draw the architecture of a modern digital company, it is going to have a streaming platform at the center.
12/ Here was the history of how we went about building Kafka into to what I think is a complete platform for streaming:
13/ We actually started by thinking about stream *processing*, but realized you need to read/write/store streams before you need processing.
14/ There'd been stream processing startups in the 00s and they failed. Why? Companies didn't have streams accessible for processing!
14/ So the 1st step was implementing a log-like abstraction for continuous streams and being to run that at company scale w/ pub/sub APIs.
15/ Next we made this fault-tolerant and replicated so it could be depended on as a store of data, not just a transient queue.
16/ The importance of storage is often missed. But it is critical for event stream processing to work in practice. confluent.io/blog/okay-stor…
17/ Next feature was implementing compacted topics, a sleeper feature, most people didn't get it at first. kafka.apache.org/documentation/…
18/ But it's a big deal--it let us have an immutable event stream representation of tables of mutable data that continually evolve/update.
19/ You can subscribe to these evolving tables and process them to create "materialized views" that are continually updated.
20/ Joining together pure events (what's happening in the world) & tables (current state of the world) is at the heart of stream processing
21/ After that we added Connector and Stream Processing APIs. Our goal was to make programs that use streams *easy*.
22/ These APIs have been a huge hit because you can quickly connect up streams from things you have and build apps on top of them.
23/ Most recently we added Transactions to Kafka. confluent.io/blog/exactly-o…
24/ This is a really fundamental thing that gives a kind of "closure property" for streaming transformations.
25/ This makes it really easy to build scalable, fault-tolerant, stateful apps that process event streams and have these get correct results
26/ People mostly know this from Kafka's streams API, but really this is just one instantiation in Java.
27/ Kafka's support for stream processing is primarily a protocol-level capability that can be represented in any language.
28/ This is an important distinction. Stream processing isn't one interface, the idea you could build it as a single java library is silly.
29/ There are many ways to express continual programs: sql, function-as-a-service, collection-like DSLs, many programming languages.
30/ A protocol is the right way to support this kind of diversity.
31/ We want Kafka to support a rich ecosystem of these, well beyond what we could ever build in a single project (and it already does).
32/ So that is the vision: the ability to read, write, and process streams of data with transactional correctness at company-wide scale.
32/ And that concludes this short (!) summary of why it took us so long.
33/ People say the infrastructure world is moving very fast, I think the exact opposite. These projects take a decade of focused effort.
34/ And we're not done. :-)
