Discover and read the best of Twitter Threads about #systemDesign
Most recents (17)
How Canva efficiently removes duplicates from billions of images
T๐๐ถ๐ป ๐ง๐ฟ๐ผ๐๐ฏ๐น๐ฒ๐: ๐๐ผ๐ ๐๐ฎ๐ป๐๐ฎ ๐๐ผ๐ป๐พ๐๐ฒ๐ฟ๐ ๐๐ถ๐น๐น๐ถ๐ผ๐ป๐ ๐ผ๐ณ ๐๐บ๐ฎ๐ด๐ฒ๐ ๐๐ถ๐๐ต ๐๐ณ๐ณ๐ถ๐ฐ๐ถ๐ฒ๐ป๐ฐ๐
๐ฏย #Refactor75ย โกDay41 #systemdesign #content #engineers
T๐๐ถ๐ป ๐ง๐ฟ๐ผ๐๐ฏ๐น๐ฒ๐: ๐๐ผ๐ ๐๐ฎ๐ป๐๐ฎ ๐๐ผ๐ป๐พ๐๐ฒ๐ฟ๐ ๐๐ถ๐น๐น๐ถ๐ผ๐ป๐ ๐ผ๐ณ ๐๐บ๐ฎ๐ด๐ฒ๐ ๐๐ถ๐๐ต ๐๐ณ๐ณ๐ถ๐ฐ๐ถ๐ฒ๐ป๐ฐ๐
๐ฏย #Refactor75ย โกDay41 #systemdesign #content #engineers
Canva has more than 20 billion user-uploaded images, with 50 million uploaded daily.
Dealing with a large variety of media creates difficulties when it comes to moderating and minimizing unnecessary duplicate content
Dealing with a large variety of media creates difficulties when it comes to moderating and minimizing unnecessary duplicate content
To solve this problem, Canva uses perceptual hashing with an internally built reverse image search system
[1/13] As we're witnessing recurring layoffs, it's crucial to stay prepared. This thread is based on my experience of interviewing with Western companies in early 2022. I'll share insights & resources that proved invaluable. Hopefully, it aids your #InterviewPreparation journey.
[2/13] Struggling with HTTPS and digital certificates? This comprehensive guide will clear your doubts - cryptobook.nakov.com. A strong foundation in #CyberSecurity is essential in today's digital landscape.
[3/13] Mobile System Design was a challenging topic due to scarce resources. Until I found this repository - github.com/weeeBox/mobileโฆ. It's a goldmine of information for any aspiring #MobileSystemDesign enthusiast.
After months of interviewing for my next #SoftwareEngineering gig, I wanted to give my two cents on the process 1/N ๐งต
Most notable for me was how drastically different the interview experience was compared to my last time job hunting 6 years ago. 2/N
Back then, my first coding interview was to implement a roman numeral parser on a whiteboard in a language of the interviewer's choice. It was an experience that felt much closer to hazing than a relevant technical assessment. 3/N
What happens behind the scenes when we ๐ฌ๐ก๐จ๐ฉ ๐จ๐ง๐ฅ๐ข๐ง๐?
.
.
Disclaimer: I have limited knowledge of the eCommerce system. The diagram below is based on my research. Please suggest better names for the components or let me know if you spot an error.
.
.
Disclaimer: I have limited knowledge of the eCommerce system. The diagram below is based on my research. Please suggest better names for the components or let me know if you spot an error.
The diagram shows the 4 key business areas in a typical e-commerce company: procurement, inventory, eComm platform, and transportation.
1๏ธโฃ Procurement
Step 1 - The procurement department selects suppliers and manages contracts with them.
Step 2 - The procurement department places orders with suppliers, manages the return of goods, and settles invoices with suppliers.
Step 1 - The procurement department selects suppliers and manages contracts with them.
Step 2 - The procurement department places orders with suppliers, manages the return of goods, and settles invoices with suppliers.
How does Netflix scale push messaging for millions of devices?
This post draws from an article published on Netflixโs engineering blog. Hereโs my understanding of how the online streaming giantโs system works.
This post draws from an article published on Netflixโs engineering blog. Hereโs my understanding of how the online streaming giantโs system works.
๐๐๐ช๐ฎ๐ข๐ซ๐๐ฆ๐๐ง๐ญ๐ฌ & ๐ฌ๐๐๐ฅ๐
- 220 million users
- Near real-time
- Backend systems need to send notifications to various clients
- Supported clients: iOS, Android, smart TVs, Roku, Amazon FireStick, web browser
- 220 million users
- Near real-time
- Backend systems need to send notifications to various clients
- Supported clients: iOS, Android, smart TVs, Roku, Amazon FireStick, web browser
๐๐ก๐ ๐ฅ๐ข๐๐ ๐จ๐ ๐ ๐ฉ๐ฎ๐ฌ๐ก ๐ง๐จ๐ญ๐ข๐๐ข๐๐๐ญ๐ข๐จ๐ง
1. Push notification events are triggered by the clock, user actions, or by systems.
2. Events are sent to the event management engine.
1. Push notification events are triggered by the clock, user actions, or by systems.
2. Events are sent to the event management engine.
How does Disney Hotstar capture 5 Billion Emojis during a tournament?
Dedeepya Bonthu [1] wrote an excellent engineering blog that captures this nicely. Here is my understanding of how the system works.
Dedeepya Bonthu [1] wrote an excellent engineering blog that captures this nicely. Here is my understanding of how the system works.
1. Clients send emojis through standard HTTP requests. You can think of Golang Service as a typical Web Server. Golang is chosen because it supports concurrency well. Threads in GoLang are lightweight.
2. Since the write volume is very high, Kafka (message queue) is used as a buffer.
How does Twitter work? Letโs take a look at it from the architectural point of view before Elon takes it.
๐๐ก๐ ๐๐ข๐๐ ๐จ๐ ๐ ๐๐ฐ๐๐๐ญ:
1๏ธโฃ A tweet comes in through the Write API.
2๏ธโฃ The Write API routes the request to the Fanout service.
#twitter #systemdesign
๐๐ก๐ ๐๐ข๐๐ ๐จ๐ ๐ ๐๐ฐ๐๐๐ญ:
1๏ธโฃ A tweet comes in through the Write API.
2๏ธโฃ The Write API routes the request to the Fanout service.
#twitter #systemdesign
3๏ธโฃ The Fanout service does a lot of processing and stores them in the Redis cache.
4๏ธโฃ The Timeline service is used to find the Redis server that has the home timeline on it.
5๏ธโฃ A user pulls their home timeline through the Timeline service.
4๏ธโฃ The Timeline service is used to find the Redis server that has the home timeline on it.
5๏ธโฃ A user pulls their home timeline through the Timeline service.
๐๐๐๐ซ๐๐ก & ๐๐ข๐ฌ๐๐จ๐ฏ๐๐ซ๐ฒ
๐น Ingester: annotates and tokenizes Tweets so the data can be indexed.
๐น Earlybird: stores search index.
๐น Blender: creates the search and discovery timelines.
๐๐ฎ๐ฌ๐ก ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐
๐นHTTP push
๐นMobile push
๐น Ingester: annotates and tokenizes Tweets so the data can be indexed.
๐น Earlybird: stores search index.
๐น Blender: creates the search and discovery timelines.
๐๐ฎ๐ฌ๐ก ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐
๐นHTTP push
๐นMobile push
Implementing Distributed Transactions โก [in a gist]
Distributed Transactions are not theoretical; they are very well used in many systems. An example of it is 10-min food/grocery delivery.
๐
Distributed Transactions are not theoretical; they are very well used in many systems. An example of it is 10-min food/grocery delivery.
๐
โจ The UX we want is: Users should see orders placed only when we have one food item and a delivery agent available to deliver.
A key feature we want from our databases is atomicity. Our storage layer can choose to provide it through atomic operations or transactions.
๐
A key feature we want from our databases is atomicity. Our storage layer can choose to provide it through atomic operations or transactions.
๐
We will have 3 microservices: Order, Store, and Delivery.
Important decision: Store services have food, and every food has packets that can be purchased and assigned.
Hence, instead of just playing with the count, we will play with the granular food packets while ordering.
๐
Important decision: Store services have food, and every food has packets that can be purchased and assigned.
Hence, instead of just playing with the count, we will play with the granular food packets while ordering.
๐
Why is a solid-state drive (SSD) fast?
โA solid state drive reads up to 10 times faster and writes up to 20 times faster than a hard disk drive.โ [1].
โA solid state drive reads up to 10 times faster and writes up to 20 times faster than a hard disk drive.โ [1].
โAn SSD is a flash-memory based data storage device. Bits are stored into cells, which are made of floating-gate transistors. SSDs are made entirely of electronic components, there are no moving or mechanical parts like in hard drives (HDD)โ [2].
The diagram below illustrates the SSD architecture.
Step 1: โCommands come from the user through the host interfaceโ [2]. The interface can be Serial ATA (SATA) or PCI Express (PCIe).
Step 1: โCommands come from the user through the host interfaceโ [2]. The interface can be Serial ATA (SATA) or PCI Express (PCIe).
#systemdesign
What is CAP theorem?
CAP stands for Consistency, Availability, and Partition tolerance. The theorem states that you cannot achieve all the properties at the best level in a single database.
What is CAP theorem?
CAP stands for Consistency, Availability, and Partition tolerance. The theorem states that you cannot achieve all the properties at the best level in a single database.
But we can only pick two out of three at a time and that totally depends on our prioritize based on requirements.
What is Data Consistency?
Consistency means that any read request will return the most recent write.
Consistency means that any read request will return the most recent write.
#systemdesign #ConsistentHashing
Does load balancer use hashing?
Hashing is one of the common methods used in Load Balancing for distributing requests among the web servers.
Does load balancer use hashing?
Hashing is one of the common methods used in Load Balancing for distributing requests among the web servers.
How load balancer uses hashing?
We have something known as request_id that is uniquely and randomly generated. Each request by the client is uniquely identified by this request_id. Load balancer hashes the request_id with the proper hash function and distributes the requests.
We have something known as request_id that is uniquely and randomly generated. Each request by the client is uniquely identified by this request_id. Load balancer hashes the request_id with the proper hash function and distributes the requests.
But here comes the problem, if one server is added or removed from the pool of servers, we need to rehash all requests from the start and distribute them to the servers. So we can use the Consistent Hashing technique to solve this problem.
Here is a simplified design diagram that explains how a stock exchange such as Nasdaq or Newyork exchange works under the hood: #developers #systemdesign
Step 1: A client places an order via the brokerโs web or mobile app.
Step 2: The broker sends the order to the exchange.1/6
Step 1: A client places an order via the brokerโs web or mobile app.
Step 2: The broker sends the order to the exchange.1/6
Step 3: The exchange client gateway performs operations such as validation, rate limiting, authentication, normalization, etc., and sends the order to the order manager.
Step 4-5: The order manager performs risk checks based on rules set by the risk manager.2/6
Step 4-5: The order manager performs risk checks based on rules set by the risk manager.2/6
Step 6: Once risk checks are passed, the order manager checks if there is enough money in the wallet.
Step 7 - 9: The order is sent to the matching engine. The matching engine sends back the execution result if a match is found. 3/6
Step 7 - 9: The order is sent to the matching engine. The matching engine sends back the execution result if a match is found. 3/6
Daily Bookmarks to GAVNetย 12/13/2021 greeneracresvaluenetwork.wordpress.com/2021/12/13/daiโฆ
Researchers one step closer to optoacoustic endoscopic probe for microsurgery
phys.org/news/2021-12-cโฆ
#microsurgery, #EndoscopicProbe, #OptoAcousticImaging, #MedicalDiagnostics
phys.org/news/2021-12-cโฆ
#microsurgery, #EndoscopicProbe, #OptoAcousticImaging, #MedicalDiagnostics
The Overlooked Levels of the Creator Economy
future.a16z.com/creator-economโฆ
#CreatorEconomy, #CreatorTools, #MonetizationPlatforms, #CreatorLevels
future.a16z.com/creator-economโฆ
#CreatorEconomy, #CreatorTools, #MonetizationPlatforms, #CreatorLevels
10 System design concepts that can help you in coding interviews
1. Load Balancing
2. Caching
3. Data Partitioning
4. Indexes
5. Proxies
6. Redundancy and Replication
7. SQL vs. NoSQL
8. Distributed Systems
9. CAP Theorem
10.Consistent Hashing
#systemdesign #programming #design
1. Load Balancing
2. Caching
3. Data Partitioning
4. Indexes
5. Proxies
6. Redundancy and Replication
7. SQL vs. NoSQL
8. Distributed Systems
9. CAP Theorem
10.Consistent Hashing
#systemdesign #programming #design
And, if you need a resource to master system design, you can checkout Grokking the System Design on @EducativeInc
educative.io/courses/grokkiโฆ
educative.io/courses/grokkiโฆ
And, if you need more choices, I have shared few more courses here
medium.com/javarevisited/โฆ
medium.com/javarevisited/โฆ
Ever wonder how replication happens between Master and Replica? How changes on Master propagates to Replica?
This is a short thread of how it happens
#systemdesign #distributedsystems
๐งต
This is a short thread of how it happens
#systemdesign #distributedsystems
๐งต
Any write operation happening on the Master is logged in the Replication log file as an event. The format in which these events are logged in the Log file is called Replication Format.
The two common Replication formats:
- Statement-based format
- Row-based format
The two common Replication formats:
- Statement-based format
- Row-based format
โจ Statement-based Format
The Master records the operation as an event in its log, and when the Replica reads this log, it executes the same operation on its copy of data.
This way, the operation on the Master is executed on the Replica, which keeps it in sync with the Master.
The Master records the operation as an event in its log, and when the Replica reads this log, it executes the same operation on its copy of data.
This way, the operation on the Master is executed on the Replica, which keeps it in sync with the Master.
Just wrapped up my 1:1 call with one of my cohort-ian and we ended up building an infinitely scalable Distributed Task Scheduler, AWS CloudWatch Events, DKron, and Quartz Scheduler, in under 30 minutes.
When foundations are clear, no system is harder to design ๐ช
#systemdesign
When foundations are clear, no system is harder to design ๐ช
#systemdesign
The features we discussed and designed were:
- Infinite task ingestion
- 30 second SLA of execution
- Execution Framework that supports Binaries, Scripts, Remote Executions
- Fault tolerance of Scheduler Nodes
- Repeatability of tasks
- Exactly-once schedule and execution
- Infinite task ingestion
- 30 second SLA of execution
- Execution Framework that supports Binaries, Scripts, Remote Executions
- Fault tolerance of Scheduler Nodes
- Repeatability of tasks
- Exactly-once schedule and execution
The design we discussed did not just have random boxes of high-funda components but rather the actual tools and techs that we would be using, along with their pros and limitations. ๐ช
The progression from coding to designing architectures is the practice of refactoring the structure and behavior of elements, at increasingly higher levels of scale.
#softwareDesign #softwareArchitecture
#softwareDesign #softwareArchitecture
... where system elements can be
field/ var/ param -> method/ function/ procedure -> class -> aggregate/ composite/ collection -> module/ package -> application/ service -> sub-system -> system -> system of systems
#codenewbie #systemDesign
field/ var/ param -> method/ function/ procedure -> class -> aggregate/ composite/ collection -> module/ package -> application/ service -> sub-system -> system -> system of systems
#codenewbie #systemDesign
... where at each level of scale, the goal of refactoring is to win the trade-offs between time, cost, quality
#architects #management #visionary #leadership
#architects #management #visionary #leadership
