Thread by @NaveenS16

In this thread, I will give a little brief overview of the Apache Kafka platform. Like what it is, how it works, core concepts, its use cases, and the benefits of using it.

#100DaysOfCode #ApacheKafka #Kafka
🧵👇

1. What is Apache Kafka?

Apache Kafka is an open-source distributed streaming platform developed originally by LinkedIn as a messaging queue, but now Kafka is much more than a messaging queue. and later donated to the Apache Software Foundation. (1/3)

👇

(cont'd...)

The project, written in Scala and Java, aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. (2/3)

👇

(cont'd...)

In general, Kafka is used to building real-time event streaming applications. As Kafka is distributed, which means that it can be scaled up when needed. All that you need to do is to add new nodes (servers) to the Kafka cluster. (3/3)

👇

2. How does Apache Kafka Work?

The platform leverages a streaming process enabling the processing of data in a parallelly connected system. This feature allows different apps to execute the record as soon as it arrives,(1/2)

👇

(cont'd...)

instead of waiting for the output of the previous record. It stores the data in a fault-tolerant durable way. In simple terms, Kafka producer applications write data to topics, and the consumer applications read from the topic. (2/2)

👇

3. Why use Apache Kafka?

There are numerous choices when it comes to choosing to publish/subscribe to messaging systems. This begs the question as to what makes Kafka a standout choice for developers. (1/2)

👇

(cont'd...)

- Multiple producers & consumers
- Disk-Based Retention
- High performance
- Scalable (2/2)

👇

4. The real use cases of Apache Kafka

- Real-time data processing
- Application activity monitoring
- Logging and/or monitoring system
- Messaging
- Metrics & logging
- Stream processing

👇

5. 4 major Kafka components:

Topic – stream of messages belonging to the same type
Producer – that can publish messages to a topic
Brokers – set of servers where publishes messages are stored
Consumer – that subscribes to various topics & pulls data from brokers.

👇

Apache Kafka® 101: Topics

Apache Kafka®'s topic is a log of events, and a log is a simple data structure with well-known semantics.

▶️

👇

Apache Kafka® 101: Producers

A producer is an external application that writes messages to a Kafka cluster, communicating with the cluster using Kafka’s network protocol.

▶️

👇

Apache Kafka® 101: Brokers

Kafka is made up of brokers or a network of machines. These might be containers running on pods and on virtualized servers running on actual processors in a physical datacenter, rather than distinct physical servers.

▶️

👇

Apache Kafka® 101: Consumers

The consumer is an external application that reads messages from Kafka topics and does some work with them, like filtering, aggregating or enriching them with other information sources.

▶️

👇

6. Apache Kafka's has benefits above traditional messaging:

- Fast: A single Kafka broker can serve thousands of clients by handling megabytes of reads and writes/second
- Scalable: Data are partitioned and streamlined over a cluster of machines to enable larger data (1/2)

👇

(cont'd...)

- Durable: Messages are persistent and is replicated within the cluster to prevent data loss
- Distributed by Design: It provides fault tolerance guarantees and durability

That;s all for this thread and thank you for reading this thread, liking it and sharing it.