What's the secret ingredient to service integration in a distributed system?

It's choosing the right communication pattern for the requirement.

Here are 4 must-know service communication patterns:

🧵 [1] Request Response with HTTP

This is a synchronous pattern. One service sends a request to another and remains blocked until it gets a response or an error.

REST is the most popular architectural style for this pattern.

However, the pattern has one pitfall.

If you have a chain of service calls (A to B to C to D), one weak link in the chain can fail the entire operation.

Not a great experience for the user as well as the developer.

Open it for a quick crash course on Kafka.

🧵 ✅What is Kafka?

It's a distributed event store and a streaming platform.

Began as an internal project at LinkedIn.

Today, some of the largest data pipelines in the world across organizations like Netflix and Uber rely on Kafka.

If you are dealing with event-driven architecture, you've three main ways of dealing with events.

Here's a brief intro about them:

🧵 [1] Event Notifications

The event just notifies that something has occurred and the payload carries the absolute minimum state.

For example, just the ID of the entity (order ID, customer ID, and so on).

The receiving component may call the source component to fetch additional information.

Such events are general-purpose and great for broadcasting requirements.

I've been writing the System Design Codex newsletter for 47 weeks now.

Here are 23 articles from the newsletter that can help you understand a System Design concept:

🧵 [1] 3 Types of Event Patterns in EDA

Why read it: Learn the different types of event patterns and when to use them

newsletter.systemdesigncodex.com/p/3-types-of-e…

Latency is the new downtime in distributed systems.

But these 8 must-know strategies can help reduce latency.

🧵 [1] Caching

The goal of caching is to minimize costly database lookups and perform high-latency computations.

Store frequently accessed data in a cache

If you want to get better at System Design interviews, remember these 24 principles

🧵 [1] If you need to scale an individual component, stick with Vertical Scaling first. Then, look at Horizontal Scaling when needed

[2] To deal with traffic spikes, evaluate the use of Autoscaling for Dynamic Resource Allocation

[3] If the system requires High Availability with Performance, consider using a Load Balancer

10 points I wish I knew when starting as a software developer

[🧵]

[1] Don't reinvent the wheel. Leverage existing solutions when you can. [2] Working code is 25% of the game. Maintaining and evolving that code base for years is the real deal.

You can't avoid Data Structures while going for an interview.

But don't worry!

You don't need to know all of them to succeed.

These 5 data structures can help you solve 90% problems.

[🧵] #1 - Array

The most basic data structure out there.

Elements are stored in contiguous memory locations.

We can access elements in an array using the index value.

The index is the position of each element in an array.

It starts with 0 & incremented by 1 for every new element.

HTTP is a stateless protocol.

This means every request is independent...

...and doesn't relate to the previous request.

But users aren't stateless.

They don't want to log in to your application every time they make a request.

So, how can you help them?

Find out👇

[🧵] The solution is to use cookies.

A cookie, really?

Yes...but not the one you eat when you're hungry.

A cookie is basically a key-value pair that's stored on the browser.

Here's how it works...

NoSQL databases have skyrocketed in popularity.

But we use NoSQL as a common term for anything that's not SQL.

In reality, NoSQL databases come in 4 different flavors.

And each flavor has a particular use case.

Know everything about it in the next 5 tweets.🧵 #1 - Document Databases

This is probably the most popular category of NoSQL databases.

Data is stored in JSON, BSON, or XML format.

Tables = Collections
Records = Documents

Documents can closely mimic the application domain object.

Examples - MongoDB, Couchbase, etc.

The interview question with no fixed answer but endless possibilities.

What are the core concepts you consider before designing a system?

[🧵] #1 - Scalability

Let's say you run a restaurant.

And are committed to keeping the same level of service no matter how many customers turn up.

How to scale your service when the number of customers increases from 10 to 50?

Or from 50 to 100? Or from 100 to 500?

To design a system, the CAP Theorem is super-important.

But it can also be confusing.

Here's CAP Theorem explained in a way you'll NEVER forget (with Tom the Prankster)

[🧵] A system has 3 guarantees:
- Consistency
- Availability
- Partition Tolerance

Now, Partition Tolerance is a must.

From pigeons to fiber-optic cables, networks are still unreliable.

That leaves you with Consistency or Availability.

Why?

Let's find out with Tom the Prankster.

In mid-2020, Notion came under an existential threat.

Why?

Their Monolithic PostgreSQL DB faced a catastrophic situation.

The same DB had served them 5 years & 4 orders of magnitude growth.

But it was NO longer sufficient.

Here's the story of how Notion was saved🧵 #1 - The First Signs of Trouble

- Frequent CPU spikes in the DB nodes
- Migrations became unsafe & uncertain
- Too many on-call requests for engineers

The monolithic database was struggling to cope with Notion's tremendous growth.

Happy problems!

But things turned ugly👇

Uber Eats handles millions of product images every hour.

At this scale, duplicate images are a big problem.

Storing the same image multiple times = huge drag on processing & CDN costs.

So, how does Uber deal with duplicate images?

Find out in the next 6 tweets 🧵 #1 - Initial Implementation

The first solution was quite basic.

There were 3 steps - download, process & store the image URLs.

No de-duplication is involved.

Disadvantages:

✅ No reuse of images.
✅ No re-processing of updated images.

What changed?👇

How to send messages using Kafka?

The Kafka Producer provides 3 different strategies to send messages from your application to the broker.

Learn about them in just 5 Tweets 🧵 #1 - Fire and Forget

Send a message to the Broker & forget about it.

No acknowledgment. Nothing.

Kafka is Highly-Available. So big chance that the messages will be delivered.

But you can't rule out lost messages.

Software release days are tense nail-biting affairs.

Every team member is on the edge of their seat.

A failed release impacts all roles whether manager or developer.

But why are releases so tough?

Because of 3 Anti-Patterns of Deploying Software

Find out more & solutions 🧵 #1 - Manual Deployment.

One of the biggest sins in 2023.

But everyone has done it at least once!

Tell-tale signs:
- Extensive document with steps
- Relying on experts
- Unpredictable outcomes post-release
- Rollbacks
- 3 AM support calls

Solution - Automatic Deployments

How to make consistent $$ as a Tech Blogger?

This thread 🧵 describes the complete process that worked for me.

- Content
- Platforms
- Revenue

No spending on Ads.

No algorithm hacking.

Just plain organic reach over time.

Only 20 tweets to help you Diversify Your Income👇 #1 - Choose a Tech Discipline

Introspect which tech area sparks the greatest curiosity within you.

Curiosity alone will make you continue in tough times.

Examples:
- Web Dev
- Front-End
- Backend
- DevOps
- IaC
- Cloud

Don't choose something because everyone else is.

Kafka is super popular among developers & large orgs.

Began as an internal project at LinkedIn.

Grew rapidly.

Today, the largest data pipelines in the world use Kafka.

Orgs like Netflix & Uber rely on it for their workflows.

Get a helicopter view of Kafka (Just 9 Tweets)🧵 #1 - Kafka Messages & Batches

Basic Unit of data in Kafka = Message

A Message is like a record in a DB table. Sent as an array of bytes.

Messages are grouped into batches.

The batch approach = increased throughput.

Once a batch is ready, it's sent to a Topic & Partition👇

Don't be a code monkey.

A code monkey is an expendable programmer.

No fun in being that!

You must strive to increase your value as a developer.

One of the best ways is understanding the WHY of your code.

The next 8 tweets will convince you of this fact.🧵 #1 - Context

Building a software system = solving a puzzle.

Each functionality = a piece achieving a business goal.

Coding without knowing the goal = a translator job.

Stop being a translator! Even ChatGPT can do that.

Understand the business goal before writing code.

Scaling databases can be hard.

Sharding is one of the ways to do so. But it is also complex.

However, many big products like Discord, Notion & Quora use sharding to scale their databases.

But what exactly is Sharding and how is it so complex?

Find out in these 8 tweets:🧵 #1 - What is Sharding?

It's a DB architecture pattern that evolves from horizontal partitioning.

You separate the rows of one table into multiple tables called partitions.

The aim is to reduce the data that needs to be searched for a query.

But there are 2 types of sharding👇

You think monolithic applications are hard!

You are in for a bigger surprise with Microservices.

Much of the complexity in microservices is due to the data.

If you can manage the data, you can reap the benefits.

Here are 5 patterns for data management in microservices.🧵 #1 - Database Per Service

In this pattern, each microservice is like a self-sufficient island🏝

It has its own logic and manages its own data.

No direct access to this data without going through the API layer.

👍Scalability, Loose Coupling
👎Complex Transactions