Read on Twitter
My data eng boot camp curricula has been updated:
- Week 1: Dimension Data Modeling
Day 1: understanding dimensions. Daily dimensions vs SCDs. How to pick SCD type 1/2/3
Day 2: applied dimension data modeling. backfilling SCD tables. Incremental building SCD tables.
1/8
- Week 1: Dimension Data Modeling
Day 1: understanding dimensions. Daily dimensions vs SCDs. How to pick SCD type 1/2/3
Day 2: applied dimension data modeling. backfilling SCD tables. Incremental building SCD tables.
1/8
- Week 2: Fact Data Modeling
Day 1: understand facts. denormalized facts vs normalized facts. How to collaborate and get logging right.
Day 2: applied fact data modeling. Reduced facts for efficient long-term analysis. Cumulative table design for efficient fact analysis. 2/8
Day 1: understand facts. denormalized facts vs normalized facts. How to collaborate and get logging right.
Day 2: applied fact data modeling. Reduced facts for efficient long-term analysis. Cumulative table design for efficient fact analysis. 2/8
- Week 3: Spark
Day 1: understanding when to use Spark. Deep dive into Spark architecture and bottlenecks
Day 2: applied Spark.
understand the parallelism vs network overhead. SparkSQL vs DataFrame vs Dataset.
I’ll lead Scala Spark and @ADutchEngineer will lead PySpark
3/8
Day 1: understanding when to use Spark. Deep dive into Spark architecture and bottlenecks
Day 2: applied Spark.
understand the parallelism vs network overhead. SparkSQL vs DataFrame vs Dataset.
I’ll lead Scala Spark and @ADutchEngineer will lead PySpark
3/8
- Week 4: Flink
— Day 1: understanding Flink. When to choose streaming vs batch. Deep dive into how Kafka works.
— Day 2: applied Flink.
create sessions to understand user behavior in real time. Learning about windows, sinks, and how to build real-time analytics. 4/8
— Day 1: understanding Flink. When to choose streaming vs batch. Deep dive into how Kafka works.
— Day 2: applied Flink.
create sessions to understand user behavior in real time. Learning about windows, sinks, and how to build real-time analytics. 4/8
- Week 5: Data Quality
Day 1: proactive vs reactive data quality. design idempotent pipelines that are unit tested in Spark. Catchnerrors in development vs in production
Day 2: production data quality checks deep dive. write-audit-publish pattern vs signal table pattern. 5/8
Day 1: proactive vs reactive data quality. design idempotent pipelines that are unit tested in Spark. Catchnerrors in development vs in production
Day 2: production data quality checks deep dive. write-audit-publish pattern vs signal table pattern. 5/8
- Week 6: Data impact and storytelling
Day 1: learning how to make a case for a new data set to be created. How evangelize the data sets you’ve already created.
Day 2: data viz deep dive. How to build fast viz with pre-computed aggregates, GROUPING SETS and good design 6/8
Day 1: learning how to make a case for a new data set to be created. How evangelize the data sets you’ve already created.
Day 2: data viz deep dive. How to build fast viz with pre-computed aggregates, GROUPING SETS and good design 6/8
- Bonus Week 7: ChatGPT-driven data engineering
— Day 1: learn how to reduce your workload by 70-80% by leveraging ChatGPT to write most of your queries and pipelines for you!
— Day 2: party with ChatGPT because now you’re an amazing data engineer! 7/8
— Day 1: learn how to reduce your workload by 70-80% by leveraging ChatGPT to write most of your queries and pipelines for you!
— Day 2: party with ChatGPT because now you’re an amazing data engineer! 7/8
The first iteration of this boot camp is full! If you’re interested in joining future iterations please subscribe to my newsletter at zachwilson.tech
#dataengineering
#dataengineering
• • •
Missing some Tweet in this thread? You can try to
force a refresh
