Updated definition:

📉 Monitoring is for running and understanding other people's code (aka "your infrastructure")

📈 Observability is for running and understanding *your* code -- the code you write, change and ship every day; the code that solves your core business problems.
Questions monitoring tools (like datadog, signalfx) can answer:

* When will my disk fill up?
* Am I running out of capacity in $(cluster)?
* Did the % free memory drop after my last deploy?
* What is the avg, 90th, 99th percentile latency per service?
Questions observability tools (like honeycomb, lightstep) can answer:

* What (1..many) things do all the errors in that spike have in common?
* How many exports per second is $app doing, and how large are they, and how does this compare to the average export size in kB?
* Break down by app and sort by export size: what are your top 3 export users, and what is the sum of their total throughput compared to the overall throughput?
* Are the errors evenly distributed across workers, AZs, instance types, software versions, build_id versions, shards?
* Are the timeouts happening for all our users, or only the test users, or only our top users by write volume, etc?
* For all of the deliveries that failed over a specified time period, what are the top three reasons they failed, and what % of failures were from a single app?
Running infrastructure means running black boxes. You may have some insight into them (god i hope so) but you don't have the ability to tweak their instrumentation, and you certainly aren't shipping code changes every day.

Infrastructure code gets package-upgraded, rarely.
And when it comes to monitoring and understanding your infrastructure, metrics-based monitoring tools that let you understand performance in aggregate are the tool for the job.

Esp when workloads are high throughput with little differentiation (routers, etc) metrics are king.
When it comes to aligning developer perspective with user experience to provide core business value, though: event-based observability tools are the only way to get at the information you need.

You need the flexibility and precision of a scalpel, not an axe.
To see an expert yet beginner-friendly (and entertaining!) intro to observability for business problems, check out this talk from @seebails -- observe2020.io/2020/03/chris-…
And to continue my killjoy track record of stiffly caring about technical definitions for technical terms, if you'd like to read more, please read my three year history of observability in the software domain. aka how we got here and where we're going:

annnd -- should you be in the mood to build an o11y tool in-house, or want to argue with me about why datadog and signalfx and their ilk are definitely not observability tools (or about what constitutes an o11y tool), do read this:


• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Charity Majors

Charity Majors Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @mipsytipsy

18 Sep
"Log Everything" is:

a) logically impossible
b) a bottomless money pit
c) mostly junk (boring things being infinitely more common than interesting things)
d) ends up with a heavy bias towards errors, and
e) causes good people to lurch desperately towards bad ideas (like "AI")
a piece i wrote years ago, just as relevant today: honeycomb.io/blog/lies-my-p…
just a few minutes ago i was having a conversation with @paulosman about the future of our profession as systems engineers, and he pulled up a quote about four ways to solve problems: solution, resolution, absolution, and dissolution. (e.g. squiretothegiants.com/2016/06/16/so-…)
Read 22 tweets
9 Sep
this post is so fucking good i spent 30 minutes trying to find it again after my physical therapy appt.

"manager" is a relic term for an outdated role. modern managers can take many forms:

* master artisans
* outcome owners
* coaches
* change vanguards

the role of "management" was designed to handle workers assumed to be "lazy and uneducated, mere cogs in a machine". too many orgs perpetuate this philosophy into modern workforces, separating the doers from the decision-makers.
anyone who cares a smidge about their work craves autonomy, which means we have set up mechanisms to drive our top performers relentlessly out of practicing their craft and into a less-fulfilling role that lets them exercise some decision-making authority.
Read 8 tweets
9 Sep
It's a reasonable question, and I will attempt to answer it, but only if you promise to remember that "leadership training" itself is a sloshy pile of almost complete and total garbage. It does not "train" you to be a better "leader". It barely even pretends to try.
If I could take the same money and order everyone to therapy to talk about their childhoods instead, I would in a hot second.

(unfortunately this is "frowned on" by the nanny state and other such maskwearing meddlers 🙃)
When such leadership training is effective, it is usually to the extent that it resembles a weak-tea version of therapy, and forces you to regularly reflect on the emotional states of yourself and your teams, and talk thru your shit with a neutral party.

Which is not nothing!
Read 12 tweets
8 Sep
I keep talking to engineers who are antsy to get to senior and beyond, frustrated that it is taking so long. And I've encountered one very, very widespread blind spot around leveling

Which is ...✨not every opportunity exists✨at every company✨at every time.✨
I mean ok; if you're a junior eng you should be able to get to intermediate, pretty much anywhere. But it gets progressively trickier after that.

Even just trying to go from intermediate to senior can get interesting. How many other intermediate or senior engineers are there?
Typically you will get promoted to the next level after you have already been performing at it for six months or the span of a review period or two. So how many other people also require promo-worthy, solidly-senior projects?

Does your ladder include mentoring/leading others?
Read 27 tweets
6 Sep
julian's response was so on point, i wrote a whole nother piece in response to him. the argument for why management really shouldn't be a promotion, how to sell it to your org, and why to try. charity.wtf/2020/09/06/if-…
i've seen a lot of people address this question from the perspective of the newly enmanagered person, who needs to learn a new set of skills; but never from the perspective of why this is better for the org as a whole, or dwelling on the emotional fallout of going back and forth.
i've also seen a lot of people chirp "management is not a promotion, it's a change of career" whilst merrily ensconced in organizations that clearly, unmistakably treat management as a promotion. which bugs me. 🙃
Read 10 tweets
4 Sep
It blew my mind when I learned that most C-levels have a tighter relationship with their vendors than their own senior ICs.

Because engineers come and go, but vendors are forever. :mindblown:
It does explain how vulnerable they are to the anti-instrumentation sales pitch.

The idea that you might not have to have anyone who deeply understands your systems? That you can pay them $$$$ and they will autoinstrument your code and tell you what to look at? So, so tempting.
WRT instrumentation: we can make it easier, we can gather a ton of stuff up automatically, we can write libraries to standardize and enable and more.

But auto-instrumentation is exactly as useful and as usable as auto-generated commenting for your code.
Read 9 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!