Thread by @Aurimas_Gr

What does good Model Tracking System look like?

Find out in the Thread 👇

--------

𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and hit 🔔 to 𝗟𝗲𝘃𝗲𝗹 𝗨𝗽 in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space!

It should be composed of two integrated parts: Experiment Tracking System and a Model Registry.

From where you track ML Pipeline metadata will depend on MLOps maturity in your company.

If you are at the beginning of the ML journey you might be:

👇

1️⃣ Training and Serving your Models from experimentation environment - you run ML Pipelines inside of your Notebook and do that manually at each retraining.

If you are beyond Notebooks you will be running ML Pipelines from CI/CD Pipelines and on Orchestrator triggers.

👇

In any case, the ML Pipeline will not be too different and a well designed System should track at least:

👇

2️⃣ Datasets used for Training Machine Learning Models in Experimentation or Production ML Pipelines. Here you should also track your Train/Test splits. At this stage you should also save all important metrics that relate to Datasets - Feature Distribution etc.

👇

3️⃣ Model Parameters (e.g. model type, hyperparameters) together with Model Performance metrics.
4️⃣ Model Artifact Location.

👇

5️⃣ Machine Learning Pipeline is an artifact itself - track information about who and when triggered it. Pipeline ID etc.
✅ Code: Everything is code - you should version and track it.

👇

When a Trained Model Artifact is saved to a Model Registry there should always be a 1 to 1 mapping of previously saved Model Metadata to The Artifact which was outputted to The Model Registry:

👇

➡️ Model Registry should have a convenient user interface in which you can compare metrics of different Experiment versions.

👇

➡️ Model Registry should have a capability that allows change of Model State with a single click of a button. Usually it would be a change of state between Staging and Production.

👇

Finally:

6️⃣ Model Tracking System should be integrated with the Model Deployment System. Once a model state is changed to Production, Deployment Pipeline is triggered - new model version is deployed, old one - decommissioned.

👇

Model Tracking System containing these properties helps in the following way:

➡️ You will be able to understand how a Model was built and repeat the experiment.
➡️ You will be able to share experiments with other experts involved.

👇

➡️ You will be able to perform rapid and controlled experiments.
➡️ The system will allow safe rollbacks to any Model Version.
➡️ Such a Self-Service System would remove friction between ML and Operations experts.

👇

👋 I am Aurimas.

I will help you Level Up in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and hit 🔔

Join a growing community of 3000+ Data Enthusiasts by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: swirlai.substack.com/p/sai-10-airfl…