Thread by @lak_gcp

Many data engineers and CIOs tend to underestimate an ironic aspect of a dramatic increase in data volumes.

The larger the data volume gets, it makes more and more sense to process the data *more* frequently!
🧵

To see why, say that a business is creating a daily report based on its website traffic and this report took 2 hours to create.

If the website traffic grows by 4x, the report will take 8 hours to create. So, the tech people 4x the number of machines.

This is wrong-headed!

2/

Instead, consider an approach that makes the reports more timely:

* Compute statistics on 6 hours of data 4 times a day
* Aggregate these 6 hourly reports to create daily reports
* You can update your "daily" report four times a day.
* Data in report is only 6 hrs old!

3/

The computational cost of both these approaches is nearly the same, yet, the second approach:
* reduces latency
* increases frequency
* spreads out the load
* handles spikes better

Plus, you get more timely, less stale reports! This can have huge business value.

4/5

Extrapolate this approach, and it makes sense to have a constantly updating dashboard – just provide 24h aggregates that are up-to-the-minute.

As data volumes increase, many businesses have this conversation and change from batch data processing to stream analytics.

5/5