,
15 tweets,
3 min read
Read on Twitter
Artificial Intelligence and Machine Learning will not solve your monitoring and observability challenges. Sorry, to be the bearer of bad news, but we're going to need humans to build and administer monitoring and observability tools for the remainder of my lifetime.
I've been referencing this quote quite a bit recently. I think it's very applicable. What buyers want is a system that with zero configuration tells them about problems in their environment. "Look, see that spike in that graph, why can't AI find that?"
Machines are ignorant. They have no intuition. They gain no experience. You literally have to type in detailed instructions for them to do even menial tasks. This is called coding.
It is possible to type in instructions that will have them find that spike in that graph. Once in my career, I trained my kid to look at a graph and find spikes for me. A five year old can do that.
So now that I have 10B five year olds, the problem is they will literally find every single fucking spike in your data. Did you know that there were 1000 anomalies a day in your time series data that you didn't care about because they did not cause a problem for your users?
In addition to the false positives problem, there is a second major challenge to putting ML and AI approaches to work on monitoring and observability: labeled training data sets.
AI has made some fantastic progress in the last decade. It can now autocorrect typing with near 70% accuracy (up from only 25% a decade ago!). (I made those statistics up). It can now identify a cat in any photo on the internet.
These are good applications for AI. There are millions of photos already which someone tagged "cat". 2.7M on Flickr actually: flickr.com/search/?text=c….
I can now throw a ton of compute at analyzing millions of pixels per photo and build a graph which, when navigated, will tell me indeed that there is a cat in that photo. Not perfect, but accuracy has increased where this is now a viable approach for image analysis at scale.
So if we can solve this for image recognition, which is a problem that's long eluded computer scientists until the last 5 years or so, why can't we find signals in machine data that tell us a problem is occurring or potentially predict future problems? Just use AI right?
The problem is that there is not a corpus of data at which you can train algorithms to identify problems. The symptoms of problems are often unique to that particular problem. The applications having the problems are unique to the company building them.
Your machine data, logs, metrics, traces, are pretty unique to your applications and your organization. Even if we overcome the uniqueness challenge, where is the labeled training data set that takes millions of time series and says "this time series good, this time series bad"?
There is very little data available which ties outages or user experience challenges to good or bad. And even if there were, these data sets would be in dozens, hundreds, maybe thousands of data points when data set they're searching is in the millions or billions of data points.
Thusly, we're back to problem #1. We have tried to put these techniques to work and what ends up is a ton of false positives. This error might be a problem, it might not, because machines have no intuition and no experience.
It is nearly impossible to imbue an algorithm with *judgement*. And ultimately, we are paying operators of complex systems for their judgement. When to page out, when to escalate, when to bring in the developers. No algorithm is going to solve that for you.
