Thread by @DVCorg

Here's a comprehensive thread on how to build a seamless batch-scoring experience.

You will learn how to:

🤖 Design ML pipelines with @DVCorg

🐈 Training with @gitlab and CML

🧪 ML experiments with @Iterativeai Studio

🚢 Deployments with @ApacheAirflow

🧵 [1/6]

🤖 Design #ML pipelines using #DVC! Streamline your process with the following steps:

1️⃣ Data prep

2️⃣ Feature engineering

3️⃣ Model training

4️⃣ Model evaluation

DVC optimizes run time & tracks changes for a more efficient workflow!

Get started with 'dvc exp run' 🚀

🧵 [2/6]

🐈 Training with @gitlab and CML

🚀 Set up train jobs using #GitLabCI & #CML! Trigger training jobs on new Merge Requests or commits with [exp] tags.

Achieve 2 goals:

1️⃣ Control experiment triggering based on code/params changes

2️⃣ Ensure merges into the main branch link to the latest model

Supercharge your workflow with DVC & CML!

🧵 [3/6]

🧪 ML experiments with @Iterativeai Studio with features such as:

✅ Automated experiment launches on code changes

✅ Version tracking for code & artefacts

✅ Easy-to-manage development process

✅ UI for launching new experiments

✅ Collaborate with the team

#DVC

🧵 [4/6]

🚢 Deployments with @ApacheAirflow

@ApacheAirflow helps to schedule and run pipelines (DAG) for various data engineering and machine learning purposes.

We will need to use the 'run_scoring' BashOperator for our use case.

🧵 [5/6]

Thanks for reading through this post! 🎉

Thanks to @mnrozhkov for creating a follow-through blog on this ❤️

You may check it out here 👇

iterative.ai/blog/automate-…

Follow @DVCorg

Congratulations!!!

You just learnt how to:

- Set up Git repo.

- Connect it to the GitLab repo.

- Use Airflow to create DAG for batch-scoring app.

- Use DVC for dataset & model versioning.

- Use CML for experiment automation.

- Set up a production environment for real-time model deployment.