,
6 tweets,
2 min read
Read on Twitter
This is a good thread by @andyfleener. Thought I’d share another example in my own thread.
We have monitors in our “war room” that cycle through our dashboards. We also use that room for regular meetings. But we don’t normally look at those dashboards unless that alert has gone off (there are no such monitors in our bullpen).
One of the graphs in the dashboard shows RPC error rates for a number of different services: it’s almost like a distribution of error rates across services, plotted over time.
One day, we were in a meeting, and I saw one of the graphs that shows RPC error rates jump up for one particular service. I called attention to it, and an experienced SRE on the team said “No, that’s normal, sometimes that service has a jump in errors”.
Another day, another regular meeting in that room, one of the SREs on my team happens to notice on that same graph that the error rates had jumped up. This time it was a different service. That one was really important. He paged the on-call (me) as soon as he saw it. It was real.
The pattern on the graph was the same for the two cases, but the name of the relevant service was key context information for identifying whether this was a potentially serious problem. fin
