Thread by @sunilc_

Messaging Queues are one of the critical components in building distributed & scalable applications.

Let's talk about them in this thread:

- what is it ?
- how to use it ?
- benefits of using it in an architecture.

🧵👇🏻

Queue:

As you might already know a queue is a data structure which works in first-in first-out manner.

It basically contains a sequence of objects that can be read and processed in the same order in which they were received.

👇🏻

Message:

A message is any data which can be in different formats like text, json, xml etc depending your application needs.

It contains all the information that needs to be sent from sender application to the receiver application.

👇🏻

For example: If we're building a system to send emails asynchronously, this is how a message looks like:

{
"email_from": "[e-mail]",
"email_to": "[e-mail]",
"subject": "Test Message",
"mail": "This is test email body."
}

👇🏻

Thus, a messaging queue provides an asynchronous protocol between sender and receiver applications.

Sender application send msgs to the queue and does not wait for any immediate response. Receiver application connects to this queue and receives and processes the messages.

👇🏻

Decoupling and Scalability:

👇🏻

A decoupled system is achieved when two or more systems can work independently without being connected to each other.

Separating out and decoupling systems like this helps us to maintain and scale them easily.

👇🏻

If the receiver applications is down, the sender application can still continue to serve the requests and push messages in to the queue.

Once the receiver application comes back up again, it can pickup and process the pending messages.

👇🏻

Advantages of a decoupled system:

- can evolve independently
- can be scaled independently
- can be written in different languages
- can be managed by different teams

👇🏻

Let's discuss a use case:

Imagine we have a service which receives thousands of requests per second to generate invoice pdfs for your customers.

In this case we can have a main API service which handles the requests,a queue and a worker service which processes the messages.

👇🏻

In this case the API service doesn't need to have the logic of all the calculations and generating the PDFs. It would just insert a message into the queue for every invoice generation request and return a response.

👇🏻

Worker service then picks up the messages and generates invoices for individual requests and then send the invoice PDFs to your customers.

In future if we want to scale the application to generate more invoices per second, we can just add more workers to the worker service.

👇🏻

Even if the processing of a particular message fails, we can add retry logic and delete the message from queue only when it is processed successfully.

👇🏻

If you want to learn cloud computing concepts like these which are very important to know to designing systems, I recommend buying @dvassallo's AWS course:

gumroad.com/a/238777459/Ms…