Thread by @Aurimas_Gr

What are 𝗟𝗮𝗺𝗯𝗱𝗮 𝗮𝗻𝗱 𝗞𝗮𝗽𝗽𝗮 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀?

🧵

#Data #DataEngineering #MLOps #MachineLearning #DataScience

Lambda and Kappa are both Data architectures proposed to solve movement of large amounts of data for reliable Online access.

👇

The most popular architecture has been and continues to be Lambda. However, with Stream Processing becoming more accessible to organizations of every size you will be hearing a lot more of Kappa in the near future. Let’s see how they are different.

👇

𝗟𝗮𝗺𝗯𝗱𝗮.

➡️ Ingestion layer is responsible for collecting the raw data and duplicating it for further Real Time and Batch processing separately.
➡️ Consists of 3 additional main layers:

👇

👉 Speed or Stream - Raw Data here is coming in Real Time and is processed by a Stream Processing Framework (e.g. Flink) then passed to the Serving layer to create Real Time Views for low latency near to Real Time Data access.

👇

👉 Batch - Batch ETL Jobs with batch processing Frameworks (e.g. Spark) are run against raw Data to create reliable Batch Views for Offline Historical Data access.

👇

👉 Serving - this is where the processed Data is exposed to the end user. Latest Real Time Data can be accessed from Real Time Views or combined with Batch Views for full history. Historical Data can be accessed from Batch Views.

👇

❗️ Processing code is duplicated for different technologies in Batch and Speed Layers causing logic divergence.
❗️ Compute resources are duplicated.

👇

❗️ Need to manage two Infrastructures.
✅ Distributed Batch Storage is reliable and scalable, even if the System crashes it is easily recoverable without errors.

👇

𝗞𝗮𝗽𝗽𝗮.

➡️ Treats both Batch and Real Time Workloads as a Stream Processing problem.
➡️ Uses Speed Layer only to prepare data for Real Time and Batch Access.
➡️ Consists of only 2 main layers:

👇

👉 Speed or Stream - similar to Lambda but (optionally) often contains Tiered Storage which means that all of Data coming into the system is stored indefinitely in different Storage Layers. E.g. S3 or GCS for historical data and on disk log for hot data.

👇

👉 Serving - same as Lambda but all transformations that are performed in Speed Layer are never duplicated in Batch Layer.

❗️ Some transformations are hard to perform in Speed Layer (e.g. complex joins) and are eventually pushed to Batch storage for implementation.

👇

❗️ Requires strong skills in Stream Processing.
✅ Data is processed once with a single Stream Processing Engine.
✅ Only need to manage single set of Infrastructure.

👇

👋 I am Aurimas.

I will help you Level Up in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and hit 🔔

Join a growing community of 5100+ Data Enthusiasts by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: swirlai.substack.com/p/sai-15-whats…

Also join the SwirlAI Data Talent Collective that will help you find your next career step: swirlai.pallet.com/talent/welcome…