LinkedIn sharing their streaming architecture at #strangeloop 
Explaining the importance of my favorite pattern: stream-table join.
Making aggregation more reliable by adding a durable stream between stages.
Two dimensions of scale: ingestion scale and processing scale. They must be dealt with separately.
One slide for Kafka at scale:
Why Kafka ops is hard... Rebalancing is key to running at scale.
Data access is the bigger scalability problem because unlike CPU, it's hard to just add more resources
If you use remote tables, you need to worry not to DDos your own database.
Better option: use CDC to create an internal cache. No remote access and no DDos - all lookups are local.
By adding integration with @ApacheBeam, LinkedIn hope to get python support for Samza and perhaps more languages in the future.
No stream processing is good at auto-scale. Why should Java developers worry about how many containers? Or worse, SQL developers.
Stream processing suffers from lack of tools compared to batch. We have some and need lots more.
