Thread by @mulegirl

Let's talk about being metrics-driven. If you are focused on metrics rather than learning or outcomes, you risk extreme tunnel vision.

I mean for any of us who grew up in the US and went to college, our education and capacity to learn was reduced to GPAs and board scores that determined our fates at a tender age, so it isn't surprising this mindset is appealing to any manager who did well by those numbers.

At the extreme end of managing to a metric, at all costs you get Wells Fargo.

And I think we can agree that's bad, right?

Measurements *feel* inherently objective, more so than descriptions, but they aren't necessarily.

The definition of the standard, its application, the placement of instruments, and the interpretation of the results are all based on human judgment.

Up until last year the definition of a kilogram was a golfball-sized object made of a platinum alloy machined into a cylinder and kept under a double glass bell in Saint-Cloud, France.

I only learned of this wildly Borgesian situation recently.

en.wikipedia.org/wiki/Internati…

Measurement systems have an international governing body with committees. We don't think about that too often.

"Historically the IPK has been compared to its official copies at intervals of about 40 years, with the exception of the "extraordinary campaign" carried out in 2014"

So, back to metrics. You can count the number of people who downloaded your app, the number of things you sold, your revenue.

Those are actually countable, quantitative measurements of things that happened in the world.

But they won't tell you why those numbers are what they are, which actually doesn't give you the feedback you need to take action to change them.

You need additional insight into cause and effect for that.

Sadly, understanding cause and effect is hard, especially in complex systems.

Understanding cause and effect of human behavior is extremely hard, for two reasons (at least)…

1) Human behavior is complex
2) Most of us have a really bad working model of human behavior (including our own)

This is especially true of people who want to think of themselves as particularly analytic, logical, or rational because people are not at all.

So a huge irony in tech & business is that the people who pride themselves on making decisions on "the best available data" are actually biased towards inferior data because the best, most useful information about human behavior is not quantitative.

Instead of confronting this deeply uncomfortable fact, a lot of people are running a lot of surveys that quantify responses to a deeply flawed instrument and completely gloss over the embedded leaps in logic.

Just because someone clicks on a button to indicate agreement with a statement doesn't mean that statement had ever crossed their mind before, or that the click has any relationship whatsoever to a behavior in a real-world context or situation.

Descriptions must precede measurements, always.

If you haven't accurately or effectively defined the thing in the world you want to measure, your measurements are meaningless. This means qual before quant. You won't know what kind of instrument or scale to use otherwise.

Even if you're measuring things you're certain exist and matter, you must keep looking out at the world to make sure you aren't missing what you're not measuring.

For example: all the relationships and behaviors that influence use of your site/app that your analytics don't see.

And just to bring it around to those stupid customer satisfaction/NPS surveys, they won't tell you about the 50 other such surveys your customer saw that hour, nor about the people who were so annoyed they didn't respond and just stopped using your service.