went rummaging through slack history, realized it's been three years since i first googled the definition of 'observability' and found to my shock that:
a) it *had* a technical definition
b) it was everything i had been struggling and flailing to describe for the past six months
so, i wrote up a three year retrospective of how the term has been used and misused, claimed by vendors both good and evil, as well as the technical requirements for systems observability ... and why high cardinality is at the root of everything. thenewstack.io/observability-…
warning, it's long; and possibly of interest to only like me and five other people. but the important thing to note here is this.
the reason we got attention, in a NOTORIOUSLY packed space, is because we reflected the real pain and anger that engineers like us are feeling.
so many engineers have been shelling out ungodly sums of money to newrelic and datadog etc, despite the fact that those tools increasingly do not solve their problems, and solve a little less and less every month.
we live in a high cardinality world.
we live in a world where canned answers are increasingly ineffective.
we live in a world where you must instrument with intent, not rely on magic.
we live in a world where user experience is the only metric that matters.
we live in a world of cheap hardware and expensive time.
the metrics, logs and APM providers got their start back when they did a pretty good job of solving your problems. they've been printing money left and right ever since.
they weren't designed for a distributed world, but they've been making too much money to care.
but the undercurrent of dissatisfied folks who knew they were wasting their cash, knew their problems weren't being solved, were dubious of "high cardinality is impossible" -- has grown rapidly.
you people are why we are still around today, and gaining steam fast. 💙
turns out people don't particularly care for being gaslit by their software vendors. nor told that what they want they can't have, but how would they like thousands of dashboards instead??
vendors keep papering this gap by releasing features named e.g. "Infinite Cardinality"
instead of releasing, say, actual support for high cardinality. (you are not fooling anyone)
this smug stasis seems to be what happens when a category is raking in so much money, they don't notice or care when the world has moved on.
in conclusion, thanks to everybody who has given us a try, given us your feedback.
i too aspire to print so much money i don't have to care about anyone's feelings, but until then... we'll be over here trying to build the shit you need to understand your systems. 🧸🐝
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I woke up this am, scanned Twitter from bed, and spent an hour debating whether I could stomach the energy to respond to the latest breathless fatwa from Paul Graham.
I fell asleep again before deciding; just as well, because @clairevo said it all more nicely than I would have.
(Is that all I have to say? No, dammit, I guess it is not.)
This is so everything about PG in a nutshell, and why I find him so heartbreakingly frustrating.
The guy is brilliant, and a genius communicator. He's seen more and done more than I ever will, times a thousand.
And he is so, so, so consistently blinkered in certain predictable ways. As a former fundamentalist, my reference point for this sort of conduct is mostly religious.
And YC has always struck me less like an investment vehicle, much more like a cult dedicated to founder worship.
Important context: that post was quote tweeting this one.
Because I have also seen designers come in saying lovely things about transformation and user centricity, and end up wasting unthinkable quantities of organizational energy and time.
If you're a manager, and you have a boot camp grad designer who comes in the door wanting to transform your org, and you let them, you are committing professional malpractice.
The way you earn the right to transform is by executing consistently, and transforming incrementally.
(by "futureproof" I mean "true 5y from now whether AI is writing 0% or 100% our lines of code)
And you know what's a great continuous e2e test of your team's prowess at learning and sensemaking?
1, regularly injecting fresh junior talent
2, composing teams of a range of levels
"Is it safe to ask questions" is a low fucking bar. Better: is it normal to ask questions, is it an expected contribution from every person at every level? Does everyone get a chance to explain and talk through their work?
The advance of LLMs and other AI tools is a rare opportunity to radically upend the way we talk and think about software development, and change our industry for the better.
The way we have traditionally talked about software centers on writing code, solving technical problems.
LLMs challenge this -- in a way that can feel scary and disorienting. If the robots are coming for our life's work, what crumbs will be left for you and me?
But I would argue that this has always been a misrepresentation of the work, one which confuses the trees for the forest.
Something I have been noodling on is, how to describe software development in a way that is both a) true today, and b) relatively futureproof, meaning still true 5 years from now if the optimists have won and most code is no longer written by humans.
A couple days back I went on a whole rant about lazy billionaires punching down and blaming wfh/"work life balance" for Google's long slide of loss dominance.
I actually want to take this up from the other side, and defend some of the much hated, much-maligned RTO initiatives.
I'm purposely not quote tweeting anyone or any company. This is not about any one example, it's a synthesis of conversations I have had with techies and seen on Twitter.
There seems to be a sweeping consensus amongst engineers that RTO is unjust, unwarranted and cruel. Period.
And like, I would never argue that RTO is being implemented well across the board. It's hard not to feel cynical when:
* you are being told to RTO despite your team not being there
* you are subject to arbitrary badge checks
* reasonable accommodations are not being made