๐ก๐ผ ๐๐
๐ฐ๐๐๐ฒ๐ ๐๐ฎ๐๐ฎ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด ๐ฃ๐ผ๐ฟ๐๐ณ๐ผ๐น๐ถ๐ผ ๐ง๐ฒ๐บ๐ฝ๐น๐ฎ๐๐ฒ - next week I will enrich it with the missing Machine Learning and MLOps parts!
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
๐งต
#Data #DataEngineering #MLOps #MachineLearning #DataScience
Today - letโs review it once more. It is super helpful as these kind of Data Architectures are what you will find in real life situations.
๐ฅ๐ฒ๐ฐ๐ฎ๐ฝ:
๐
๐ฅ๐ฒ๐ฐ๐ฎ๐ฝ:
๐
๐ญ. Data Producers - Python Applications that extract data from chosen Data Sources and push it to Collector via REST or gRPC API calls.
๐
๐
2. Collector - REST or gRPC server written in Python that takes a payload, validates top level field existence and correctness, adds additional metadata and pushes it into either Raw Events Topic if the validation passes or a Dead Letter Queue if top level fields are invalid.
๐
๐
3. Enricher/Validator - Python or Spark Application that validates event schema in Raw Events Topic, performs data enrichment and pushes results into either Enriched Events Topic if validation passes and enrichment succeeds or a Dead Letter Queue if any of previous failed.
๐
๐
๐ฐ. Enrichment API - API of any flavor implemented with Python that can be called for enrichment purposes by Enricher/Validator. This could be a Machine Learning Model deployed as an API as well.
๐
๐
๐ฑ. Real Time Loader - Python or Spark Application that reads data from Enriched Events and Enriched Events Dead Letter Topics and writes them in real time to ElasticSearch Indexes for Analysis and alerting.
๐
๐
๐ฒ. Batch Loader - Python or Spark Application that reads data from Enriched Events Topic, batches it in memory and writes to MinIO Object Storage.
๐
๐
๐ณ. Scripts Scheduled via Airflow that read data from Enriched Events MinIO bucket, validates data quality, performs deduplication and any additional Enrichments. Here you also construct your Data Model to be later used for reporting purposes.
๐
๐
๐ง๐ญ. A single Kafka instance that will hold all of the Topics for the Project.
๐ง๐ฎ. A single MinIO instance that will hold all of the Buckets for the Project.
๐ง๐ฏ. Airflow instance that will allow you to schedule Python or Spark Batch jobs against data stored in MinIO.
๐
๐ง๐ฎ. A single MinIO instance that will hold all of the Buckets for the Project.
๐ง๐ฏ. Airflow instance that will allow you to schedule Python or Spark Batch jobs against data stored in MinIO.
๐
๐ง๐ฐ. Presto/Trino cluster that you mount on top of Curated Data in MinIO so that you can query it using Superset.
๐ง๐ฑ: ElasticSearch instance to hold Real Time Data.
๐
๐ง๐ฑ: ElasticSearch instance to hold Real Time Data.
๐
๐ง๐ฒ. Superset Instance that you mount on top of Trino Querying Engine for Batch Analytics and Elasticsearch for Real Time Analytics.
๐
๐
๐ I am Aurimas.
I will help you Level Up in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.
๐๐ผ๐น๐น๐ผ๐ ๐บ๐ฒ and hit ๐
Join a growing community of 5200+ Data Enthusiasts by subscribing to my ๐ก๐ฒ๐๐๐น๐ฒ๐๐๐ฒ๐ฟ: swirlai.substack.com/p/sai-15-whatsโฆ
I will help you Level Up in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.
๐๐ผ๐น๐น๐ผ๐ ๐บ๐ฒ and hit ๐
Join a growing community of 5200+ Data Enthusiasts by subscribing to my ๐ก๐ฒ๐๐๐น๐ฒ๐๐๐ฒ๐ฟ: swirlai.substack.com/p/sai-15-whatsโฆ
Also join the SwirlAI Data Talent Collective that will help you find your next career step: swirlai.pallet.com/talent/welcomeโฆ
โข โข โข
Missing some Tweet in this thread? You can try to
force a refresh
ใ
