/1 How Discord Stores Trillions Of Messages
The diagram below shows the evolution of message storage at Discord:
MongoDB ➡️ Cassandar ➡️ ScyllaDB
The diagram below shows the evolution of message storage at Discord:
MongoDB ➡️ Cassandar ➡️ ScyllaDB
/2 In 2015, the first version of Discord was built on top of a single MongoDB replica. Around Nov 2015, MongoDB stored 100 million messages and the RAM couldn’t hold the data and index any longer. The latency became unpredictable.
/3 In 2017, Discord had 12 Cassandra nodes and stored billions of messages.
At the beginning of 2022, it had 177 nodes with trillions of messages. At this point, latency was unpredictable, and maintenance operations became too expensive.
There are several reasons for the issue:
At the beginning of 2022, it had 177 nodes with trillions of messages. At this point, latency was unpredictable, and maintenance operations became too expensive.
There are several reasons for the issue:
/4
- Cassandra uses the LSM tree for the internal data structure. The reads are more expensive than the writes. There can be many concurrent reads on a server with hundreds of users, resulting in hotspots.
- Maintaining clusters, such as compacting SSTables, impacts performance.
- Cassandra uses the LSM tree for the internal data structure. The reads are more expensive than the writes. There can be many concurrent reads on a server with hundreds of users, resulting in hotspots.
- Maintaining clusters, such as compacting SSTables, impacts performance.
/5
- Garbage collection pauses would cause significant latency spikes
ScyllaDB is Cassandra compatible database written in C++. Discord redesigned its architecture to have a monolithic API, a data service written in Rust, and ScyllaDB-based storage.
- Garbage collection pauses would cause significant latency spikes
ScyllaDB is Cassandra compatible database written in C++. Discord redesigned its architecture to have a monolithic API, a data service written in Rust, and ScyllaDB-based storage.
/6 The p99 read latency in ScyllaDB is 15ms compared to 40-125ms in Cassandra. The p99 write latency is 5ms compared to 5-70ms in Cassandra.
👉 Over to you: What kind of NoSQL database have you used? How do you like it?
👉 Over to you: What kind of NoSQL database have you used? How do you like it?
/8 Subscribe to our weekly newsletter to get a 𝐅𝐫𝐞𝐞 𝐒𝐲𝐬𝐭𝐞𝐦 𝐃𝐞𝐬𝐢𝐠𝐧 𝐏𝐃𝐅 (158 pages): blog.bytebytego.com
/9 I hope you've found this thread helpful.
Follow me @alexxubyte for more.
Like/Retweet the first tweet below if you can:
Follow me @alexxubyte for more.
Like/Retweet the first tweet below if you can:
https://twitter.com/alexxubyte/status/1635306946034294784
Typo: Cassandar -> Cassandra
• • •
Missing some Tweet in this thread? You can try to
force a refresh
