Thread by @rmoff

Can’t wait for this panel discussion at #current22 with ⁦@takidau⁩ ⁦@notamyfromdbt⁩ ⁦@krisajenkins⁩ ⁦@AdiPolak⁩ ⁦@esammer⁩ - it’s gonna be awesome!
(and it’s being live-streamed - make sure you tune in!)

Are we going to have batch and streaming forever, or will they converge? @esammer says at the heart of systems lambda arch will go away and kappa will eventually win out. Once in DW perhaps batch will remain for its familiarity to analytics engineers.

@notamyfromdbt - Microbatching gets used to simulate streaming but with same toolset for familiarity, but it doesn’t scale

What’s holding people back from streaming? @takidau says streaming answers everything, but the complexity is currently a challenge. Aslo low-latency isn’t always so necessary to be worth the effort

Is streaming a superset? @esammer says in theory yes. However there are some optimisations that you can do in batch that are easier than in streaming (eg checkpoints aren’t needed in batch)

@notamyfromdbt notes that there isn’t a consensus or best practice from the streaming side of things yet unlike the modern data stack on the batch side (echoing @bennstancil’s point from his talk earlier - make it boring)