Thread by @RamsesCoraspe

📍 The need for data compression📍
🧵
When we talk about computation time we are also talking about money, data compression represents the most appropriate economic way to shorten the gap between content creators and content consumers.

Compressed files are obviously smaller and it is necessary less money and time to transfer them and cost less money to store them, content creators pay less money to distribute their content, and content consumers pay less money to consume content.

On the other hand, companies in all sectors need to find new ways to control the rapid growth of their data heterogeneous generated every day, data compression and decompression techniques are one of the most viable solutions to these problems. #bigdata

In fact, data compression and decompression techniques are the DNA of many famous distributed systems (DS) and part of their success is due to the proper use of these.
#ApacheSpark
#ApacheKafka
#ApacheFlink
#ApacheBeam
#ApacheHadoop

These DS above use a hidden universe called contextual data transformations, the raw data is never compressed by a compression algorithm, it is always processed first by this previous pipeline which rearrange the symbols so they will be more sensitive to a compression algorithm.

Bzip2 doesn't have the best performance in compression and decompression times, it has a feature that makes it special, compression and decompression can be separated into blocks, which makes it powerful to be used in DISTRIBUTED systems.

I made my own version of bzip2, and I am trying to improve compression and decompression times and add more features that could be valuable for speed up these process.

coraspe-ramses.medium.com/lossless-data-…

github.com/Wittline/wbz

@threadreaderapp unroll