What's a machine learning pipeline?
Well, it turns out that many different things classify as "machine learning pipelines."
Here are five of the different "pipelines" you should be aware of: ↓
Well, it turns out that many different things classify as "machine learning pipelines."
Here are five of the different "pipelines" you should be aware of: ↓
Our first pipeline: "Data pipeline."
This goes from ingesting the data from its sources to the final destination where we will consume it.
Sometimes, the data pipeline includes transformations of that data. Sometimes it doesn't.
This leads me to the second pipeline.
This goes from ingesting the data from its sources to the final destination where we will consume it.
Sometimes, the data pipeline includes transformations of that data. Sometimes it doesn't.
This leads me to the second pipeline.
The second pipeline: "Data transformation pipeline."
"Wait, I thought this was part of the data pipeline?" You are right; sometimes it is. Sometimes it isn't.
Sometimes, you need to separate "general" transformations from use case-specific transformations.
"Wait, I thought this was part of the data pipeline?" You are right; sometimes it is. Sometimes it isn't.
Sometimes, you need to separate "general" transformations from use case-specific transformations.
The third pipeline: "Training and evaluation pipeline."
Here is where we split, train, evaluate, and deploy a machine learning model.
Sometimes we can join this one with data transformation and make a single pipeline.
Here is where we split, train, evaluate, and deploy a machine learning model.
Sometimes we can join this one with data transformation and make a single pipeline.
The fourth pipeline: "Inference pipeline."
Here is where we transform production data, run it through the model, and process the results.
(Good practice here: use the same data transformation pipeline that you built before.)
Here is where we transform production data, run it through the model, and process the results.
(Good practice here: use the same data transformation pipeline that you built before.)
The fifth pipeline: "Monitoring and maintenance pipeline."
This one is hard to find because it's indicative of mature machine learning systems (and there aren't too many of those out there.)
Goal: monitoring, retraining, and redeploying the model.
This one is hard to find because it's indicative of mature machine learning systems (and there aren't too many of those out there.)
Goal: monitoring, retraining, and redeploying the model.
We are still figuring things out.
Sometimes, we use the same term to refer to different processes that happen at different stages. It's confusing as hell!
But that's why I'm here: to break some of these things in a way you can understand.
@svpino ← This is me. Stay tuned.
Sometimes, we use the same term to refer to different processes that happen at different stages. It's confusing as hell!
But that's why I'm here: to break some of these things in a way you can understand.
@svpino ← This is me. Stay tuned.
Great point.
Validating the data is really important whenever you can't trust the source. So yes, this will be part of your pipeline.
Validating the data is really important whenever you can't trust the source. So yes, this will be part of your pipeline.
https://twitter.com/NickXiotis/status/1451538733925380100?s=20
• • •
Missing some Tweet in this thread? You can try to
force a refresh
