Thread by @DataCereal

Alright, let's live-tweet again! Super excited to live-tweet my colleague, @VertaAI's Cory Johannsen's talk about why AAP is Not the same as ML Monitoring! #MLOps databricks.com/session_na21/w…

What is APM (New Relic, DataDog, Elastic etc. etc.)? Tracking metrics about your system, aggregating, dashboarding, alerting (1/n)

APM lets you track health, perf, availability etc. the Golden signals (2/n)

APM is only as useful as the incidents it helps identify and resolve (3/n)

Alert Fatigue is for real -- whether in DevOps, healthcare (!) or submarines (4/n)

"I understand monitoring, this is easy!" But no, models are different beasts (5/n)

What makes ML Monitoring so different? For APM, the metrics are latency, throughput, errors etc. But these units are too simplistic for models and data pipelines. We need to know data quality and data drift (6/n)

What can ML Monitoring learn from APM? (7/n)

BUT! ML Monitoring is a **superset** of APM (8/n)

APM agents assume that they understand the data that you are sending them. Not so for model monitoring, only you, the model owner knows the shape of the data and what's "correct" (9/n)

Similarly, the comparison of metrics is also model-specific -- what is your reference metric? how do you compare these metrics? (10/n)

Finally, only the model owner knows *when* a change matters and when to take action (11/n)

Luckily at @VertaAI, we have been able to build on this expertise in APM to build a purpose-built model monitoring framework! (12/n)

The way you use this framework is as follows: (1) define the statistics you care about, (2) compute these statistics on your data, (3) define alerts. That's it (13/n)

Once your statistics are in the database, you can visualize! How cool is that! (14/n)

So excited to have Cory lead the development of Verta Model Monitoring and re-imagine the innovations of APM for ML! (15/n)

Check out the Community Preview for Model Monitoring here: info.verta.ai/model-monitori…

@threadreaderapp unroll