Aurimas Griciลซnas Profile picture
Dec 21 โ€ข 14 tweets โ€ข 5 min read
What does good Model Tracking System look like?

Find out in the Thread ๐Ÿ‘‡

--------

๐—™๐—ผ๐—น๐—น๐—ผ๐˜„ ๐—บ๐—ฒ and hit ๐Ÿ”” to ๐—Ÿ๐—ฒ๐˜ƒ๐—ฒ๐—น ๐—จ๐—ฝ in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space! Image
It should be composed of two integrated parts: Experiment Tracking System and a Model Registry.

From where you track ML Pipeline metadata will depend on MLOps maturity in your company.

If you are at the beginning of the ML journey you might be:

๐Ÿ‘‡
1๏ธโƒฃ Training and Serving your Models from experimentation environment - you run ML Pipelines inside of your Notebook and do that manually at each retraining.

If you are beyond Notebooks you will be running ML Pipelines from CI/CD Pipelines and on Orchestrator triggers.

๐Ÿ‘‡
In any case, the ML Pipeline will not be too different and a well designed System should track at least:

๐Ÿ‘‡
2๏ธโƒฃ Datasets used for Training Machine Learning Models in Experimentation or Production ML Pipelines. Here you should also track your Train/Test splits. At this stage you should also save all important metrics that relate to Datasets - Feature Distribution etc.

๐Ÿ‘‡
3๏ธโƒฃ Model Parameters (e.g. model type, hyperparameters) together with Model Performance metrics.
4๏ธโƒฃ Model Artifact Location.

๐Ÿ‘‡
5๏ธโƒฃ Machine Learning Pipeline is an artifact itself - track information about who and when triggered it. Pipeline ID etc.
โœ… Code: Everything is code - you should version and track it.

๐Ÿ‘‡
When a Trained Model Artifact is saved to a Model Registry there should always be a 1 to 1 mapping of previously saved Model Metadata to The Artifact which was outputted to The Model Registry:

๐Ÿ‘‡
โžก๏ธ Model Registry should have a convenient user interface in which you can compare metrics of different Experiment versions.

๐Ÿ‘‡
โžก๏ธ Model Registry should have a capability that allows change of Model State with a single click of a button. Usually it would be a change of state between Staging and Production.

๐Ÿ‘‡
Finally:

6๏ธโƒฃ Model Tracking System should be integrated with the Model Deployment System. Once a model state is changed to Production, Deployment Pipeline is triggered - new model version is deployed, old one - decommissioned.

๐Ÿ‘‡
Model Tracking System containing these properties helps in the following way:

โžก๏ธ You will be able to understand how a Model was built and repeat the experiment.
โžก๏ธ You will be able to share experiments with other experts involved.

๐Ÿ‘‡
โžก๏ธ You will be able to perform rapid and controlled experiments.
โžก๏ธ The system will allow safe rollbacks to any Model Version.
โžก๏ธ Such a Self-Service System would remove friction between ML and Operations experts.

๐Ÿ‘‡
๐Ÿ‘‹ I am Aurimas.

I will help you Level Up in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

๐—™๐—ผ๐—น๐—น๐—ผ๐˜„ ๐—บ๐—ฒ and hit ๐Ÿ””

Join a growing community of 3000+ Data Enthusiasts by subscribing to my ๐—ก๐—ฒ๐˜„๐˜€๐—น๐—ฒ๐˜๐˜๐—ฒ๐—ฟ: swirlai.substack.com/p/sai-10-airflโ€ฆ

โ€ข โ€ข โ€ข

Missing some Tweet in this thread? You can try to force a refresh
ใ€€

Keep Current with Aurimas Griciลซnas

Aurimas Griciลซnas Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Aurimas_Gr

Dec 23
If I could only choose 5 books to read in 2023 as an aspiring Data Engineer these would be them in a specific order:

Read on in the Thread ๐Ÿ‘‡

--------

Follow me and hit ๐Ÿ”” to ๐—Ÿ๐—ฒ๐˜ƒ๐—ฒ๐—น ๐—จ๐—ฝ in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space! Image
1๏ธโƒฃ โ€๐—™๐˜‚๐—ป๐—ฑ๐—ฎ๐—บ๐—ฒ๐—ป๐˜๐—ฎ๐—น๐˜€ ๐—ผ๐—ณ ๐——๐—ฎ๐˜๐—ฎ ๐—˜๐—ป๐—ด๐—ถ๐—ป๐—ฒ๐—ฒ๐—ฟ๐—ถ๐—ป๐—ดโ€ - A book that I wish I had 5 years ago. After reading it you will understand the entire Data Engineering workflow. It will prepare you for further deep dives.

๐Ÿ‘‡
2๏ธโƒฃ โ€๐—”๐—ฐ๐—ฐ๐—ฒ๐—น๐—ฒ๐—ฟ๐—ฎ๐˜๐—ฒโ€ - Data Engineers should follow the same practices that Software Engineers do and more. After reading this book you will understand DevOps practices in and out.

๐Ÿ‘‡
Read 9 tweets
Dec 22
What is a ๐—™๐—ฒ๐—ฎ๐˜๐˜‚๐—ฟ๐—ฒ ๐—ฆ๐˜๐—ผ๐—ฟ๐—ฒ and why is it such an important element in ๐— ๐—Ÿ๐—ข๐—ฝ๐˜€ ๐—ฆ๐˜๐—ฎ๐—ฐ๐—ธ?

Find out in the Thread ๐Ÿ‘‡

--------

๐—™๐—ผ๐—น๐—น๐—ผ๐˜„ ๐—บ๐—ฒ and hit ๐Ÿ”” to ๐—Ÿ๐—ฒ๐˜ƒ๐—ฒ๐—น ๐—จ๐—ฝ in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space! Image
Feature Store System sits between Data Engineering and Machine Learning Pipelines and it solves the following issues:

โžก๏ธ Eliminates Training/Serving skew by syncing Batch and Online Serving Storages (5)

๐Ÿ‘‡
โžก๏ธ Enables Feature Sharing and Discoverability through the Metadata Layer - you define the Feature Transformations once, enable discoverability through the Feature Catalog and then serve Feature Sets for training and inference purposes trough unified interface (4๏ธ,3).

๐Ÿ‘‡
Read 15 tweets
Dec 21
Do you know what CDC(Change Data Capture) is and that there are multiple ways to implement it?

Find out in the Thread ๐Ÿ‘‡

--------

๐—™๐—ผ๐—น๐—น๐—ผ๐˜„ ๐—บ๐—ฒ and hit ๐Ÿ”” to ๐—Ÿ๐—ฒ๐˜ƒ๐—ฒ๐—น ๐—จ๐—ฝ in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space! Image
๐—–๐—ต๐—ฎ๐—ป๐—ด๐—ฒ ๐——๐—ฎ๐˜๐—ฎ ๐—–๐—ฎ๐—ฝ๐˜๐˜‚๐—ฟ๐—ฒ is a software process used to replicate actions performed against Operational Databases for use in downstream applications.

๐—ง๐—ต๐—ฒ๐—ฟ๐—ฒ ๐—ฎ๐—ฟ๐—ฒ ๐˜€๐—ฒ๐˜ƒ๐—ฒ๐—ฟ๐—ฎ๐—น ๐˜‚๐˜€๐—ฒ ๐—ฐ๐—ฎ๐˜€๐—ฒ๐˜€ ๐—ณ๐—ผ๐—ฟ CDC. ๐—ง๐˜„๐—ผ ๐—ผ๐—ณ ๐˜๐—ต๐—ฒ ๐—บ๐—ฎ๐—ถ๐—ป ๐—ผ๐—ป๐—ฒ๐˜€:

๐Ÿ‘‡
โžก๏ธ ๐——๐—ฎ๐˜๐—ฎ๐—ฏ๐—ฎ๐˜€๐—ฒ ๐—ฅ๐—ฒ๐—ฝ๐—น๐—ถ๐—ฐ๐—ฎ๐˜๐—ถ๐—ผ๐—ป (refer to 3๏ธโƒฃ in the Diagram).

๐Ÿ‘‰ CDC can be used for moving transactions performed against Source Database to a Target DB. If each transaction is replicated - it is possible to retain all ACID guarantees when performing replication.

๐Ÿ‘‡
Read 15 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(