Fred Moyer Profile picture
SLOgician. SRE @Zendesk. Engineer to engineer. Cycling, TSDBs, Dad. Always tries to have facts/math/empathy. Former DevE @circonus, Arch @turnitin

19 Nov 19, 12 tweets

.@gphat getting ready to send it with "No-Nonsense Observability Improvement" #kubecon #observability

@gphat "We are not going to define Observability today" 👍

@gphat The normal zone, the weird zone.

@gphat The robot needs to understand all of the states in the system.

@gphat Promote your work. Make sure people know about the work you are doing.

@gphat Teach basic skills. Your first interaction with tooling should not be at 3am under duress.

@gphat Stretch your systems. Resilience engineering. Chaos testing. Gamedays.

@gphat Measure usage. Capex and Opex, it costs you money either way. Observability will be one of the most expensive parts of your stack. Think about charge back to teams. At prev company, 99.8% of metrics written never got read.

@gphat Measure read usage. People are using those metrics.

@gphat Track need in detail. Mttr, mttd, big systems change a lot. Only way to learn about failure is to fail. Where are people saying "how does that work?". Measure metrics of the successful systems also. What near misses are there.

@gphat Instrument change. Big boom in conference hall. Change management, feature flags. Humans become more important when you automate things. Avoid humans being out of the loop. Instrument automation.

@gphat Stygmergy. Follow the cookies. Establish "your way". Better understanding through practice. Be prepared when finance asks about spends. Examine success as much as failure. Instrument change events.

Some of all of these should mean a more valuable Observability program.

Share this Scrolly Tale with your friends.

A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.

Keep scrolling