,
6 tweets,
1 min read
Read on Twitter
One of the things that interests me about our field is how changes in relative ratios of compute/memory/disk/network break our long-held assumptions. This one is from a recent operational experience. One of our systems sends large streams of data across the network. 1/n
To improve latency, it wraps the output stream with a GZIP compressing stream. Intuitively that makes perfect sense - the less data we have to send, the faster the transfer speed. This part of the system was last touched 5 years ago. 2/n
However, in those 5 years our datacenter network speeds increased by almost an order of magnitude, while compression speeds (which depend on single core performance) stayed relatively flat. 3/n
Today, typical gzip compression throughput (on my C5 desktop) is 0.25Gbps. Meanwhile many data center networks can do 25 or 100 Gbps, and even many Gbps per single flow. You will need compression ratios of 20 or more just to break even. 4/n
There are other compression algorithms, that offer better CPU to compression ratio trade offs. LZ4 and Snappy are good choices. And obviously economics change if you are storing compressed data long term. 5/n
But assumptions that were true 5 years ago, may no longer hold true. Keep re-evaluating them. In our case, we were able to get a good latency win. 6/6
