Thread by @jeremystan

1/ There are some great options for data warehousing today. After dozens of conversations with data teams, and a lot of testing ourselves, here is our (@eshmu and I) take on:

redshift v presto v snowflake v bigquery v athena

2/ Redshift is the incumbent. It’s a good compromise between time, value and performance.

But if your compute spikes dramatically, query times can slow to a crawl, as storage and compute are tied. Elastic resize and concurrency scaling may address this.

3/ Presto is what the most mature and technically sophisticated teams have ended up using. Your data can live anywhere, and you can spin up many isolated compute clusters as needed.

But it’s heavy lifting for configuration and management.

4/ Snowflake is like Presto but without the configuration and maintenance headaches.

But you pay a hefty bill for this, and at scale this has caused multiple organizations to consider other solutions. There is also a small fixed time per query.

5/ BigQuery is… well… Google just handles it, which is magical, and pretty cheap.

But their SQL support is bare bones (timestamps are a mess), every query has a small fixed return time, and they throttle you at ~30 simultaneous queries (you can always retry).

6/ Athena is essentially AWS “managed presto”, and is poised to compete well against Snowflake and BigQuery.

But it’s still very rough around the edges.

7/ For most use cases I would lean towards BigQuery, as it “just works” at scale, with minimal cost and complexity.

If speed is crucial, and budget is less of a concern, Snowflake could be a good fit (and works well with S3), but at scale I would run Presto.

8/ One funny complaint we’ve heard from customers switching from Redshift, is that their data warehouse bills end up much higher than forecasted.

Uncapping compute leads to more consumption, which can 2-10x the bill despite better cost / query!!