1/ Faust is a python library from for stream processing with @apachekafka from @RobinhoodApp. I think it's really cool. It highlights one of the things I think we got right with Kafka Streams: supporting stream processing in Kafka at the protocol level. github.com/robinhood/faust
2/ This means having a model in Kafka's core protocol for elastic scalability, partitioning, stateful processing, and transactionally correct processing that covers both input, output, and state changes but is decoupled from any implementation of code that does this processing.
3/ This functionality is all part of core Kafka and supported in the consumer/producer APIs (albeit in a low-level way).
Kafka Streams is really just a Java library that uses this protocol and gives reusable operators, but our hope is that will come to exist in every language.
Kafka Streams is really just a Java library that uses this protocol and gives reusable operators, but our hope is that will come to exist in every language.
4/ This is the better architecture because there isn't really one interface to rule them all for stream processing. There is lots of room for innovation here, in lots of languages, but all have to solve these core problems and should be able to exchange streams of events.
5/ Other system try to achieve language support by wrapping python around a JVM and piping data back and forth through standard in/out. That's disgusting and exactly as clunky as it sounds. You don't want a JVM strapped to the back of your little event-driven python microservice.
6/ Kafka Streams is great, but there are many possible interfaces: KSQL, FaaS, etc. I'm waiting for some to make something more like a spreadsheet (spreadsheets are a lot like stream processing!).
7/ The way Faust makes things feel Pythonic is really nice! Lot of room for other interfaces like this. What's key is that all of these interfaces can talk to each other via the same Kafka streams. A stream is a stream whether you made it with SQL, Java, or Python.
