🛠️ Tooling Tuesday 🛠️
This week: @dagsterio (dagster.io)
dagster describes themselves as a "data orchestrator for machine learning, analytics, and ETL"
Let's break that down 👇
This week: @dagsterio (dagster.io)
dagster describes themselves as a "data orchestrator for machine learning, analytics, and ETL"
Let's break that down 👇
2/ When you work with real-world data, your pipelines can get complex.
E.g., to train a language model on twitter, you might:
- Download data
- Strip out offensive tweets
- Preprocess the data
- Fit models
- Summarize training performance
- Deploy the best model to production
E.g., to train a language model on twitter, you might:
- Download data
- Strip out offensive tweets
- Preprocess the data
- Fit models
- Summarize training performance
- Deploy the best model to production
3/ In production settings, pipelines can be even more complicated.
All well and good, but doing those steps manually every time you update your model is painful, resource intensive, and hard to scale.
And what happens if you have hundreds of these pipelines you need to manage?
All well and good, but doing those steps manually every time you update your model is painful, resource intensive, and hard to scale.
And what happens if you have hundreds of these pipelines you need to manage?
4/ Enter workflow engines.
Workflow engines help you define pipelines as directed acyclic graphs (DAGs) of tasks written in any language.
They allow you to run DAGs on a schedule (or in response to triggers), monitor their execution, reproduce them, and scale them
Workflow engines help you define pipelines as directed acyclic graphs (DAGs) of tasks written in any language.
They allow you to run DAGs on a schedule (or in response to triggers), monitor their execution, reproduce them, and scale them
5/ @ApacheAirflow is the most popular workflow engine. It was developed at AirBnB in 2014 - 2015, and has become a standard since.
6/ Despite its popularity, Airflow is not universally loved.
Teams often complain that things like:
- Debugging
- Testing
- Fast local development
- Keeping track of produced data assets
are painful in Airflow
Teams often complain that things like:
- Debugging
- Testing
- Fast local development
- Keeping track of produced data assets
are painful in Airflow
7/ Airflow is a "dumb" workflow executor. It doesn't maintain state about executions, and doesn't understand data passed between steps.
This is an advantage because it makes it lightweight and general.
But @dagsterio thinks it is also the root cause of usability complaints
This is an advantage because it makes it lightweight and general.
But @dagsterio thinks it is also the root cause of usability complaints
8/ Like Airflow, @dagsterio lets you implement tasks in any language and define dependencies using python.
Unlike Airflow, Dagster:
- Types inputs and outputs
- Tracks data assets
- Separates configuration from code
- Produces execution metadata for better observability
Unlike Airflow, Dagster:
- Types inputs and outputs
- Tracks data assets
- Separates configuration from code
- Produces execution metadata for better observability
9/ Dagster calls itself a "data orchestrator" instead of a "workflow engine" because of some of these opinionated choices to improve usability in the particular case where your workflow is a data pipeline
10/ Dagster is also focused on making an easy-to-use local development experience with a nice GUI, and a seamless transition from local to remote execution.
Sounds like an upgrade! So what is the tradeoff?
Sounds like an upgrade! So what is the tradeoff?
11/ Dagster is a more structured and constrained programming model than Airflow or its other competitors.
There are more abstractions to learn when you are getting started. Usage is more rigid. E.g., some users say it's challenging to build graphs dynamically.
There are more abstractions to learn when you are getting started. Usage is more rigid. E.g., some users say it's challenging to build graphs dynamically.
12/ If you're looking for a better way to build data pipleines, @dagsterio is worth a try!
Other workflow platforms worth checking out:
- github.com/spotify/luigi
- @kubeflow pipelines
- @PrefectIO
- @pachyderminc
Other workflow platforms worth checking out:
- github.com/spotify/luigi
- @kubeflow pipelines
- @PrefectIO
- @pachyderminc
13/ What else do you like about Dagster? Any other great tools for workflow execution that we missed?
• • •
Missing some Tweet in this thread? You can try to
force a refresh
