Thread by @mipsytipsy

It is a neverending revelation to me just how little the average product engineer expected to know or care about, well ... computers?

They get very good at writing code, debugging that code, and perhaps debugging and solving problems at the intersection of code and user interactions.

But the intersection of code + architecture, data model or system resources seems to receive -- at best -- a blissful ignorance?

To clarify, I think of "product engineers" as engineers who build for external customers (who usually aren't engineers).

In contrast with infra eng, whose customers are internal users, often other engineers.

And I'm not even talking about, like, TCP packet inspection or tuning kernels or anything. I get it, not everybody's idea of a fun weekend. 🙃

But like, dumping state from a wedged process, or using gdb/strace/stap? Following and understanding the request path seems relevant.

I guess the way this manifests the most is: instrumentation. Infra engineers, in my experience, typically instrument as they write code almost automatically.

This is not because they are "better" devs, is because they are pathologically suspicious of production.

Abstractions are just like religions. They streamline complexity and provide a set of conventions that enable large groups to cooperate. They are truly wonderful things...until you start to believe in them.

And honestly the longer I do this stuff the more I think automagic instrumentation is a curse and a plague upon us.

If you get your instrumentation by installing a library and doing nothing else, you were robbed of the most necessary and irreplaceable part of the process.

Which is casting forth in time and looking at what you're building right now, and asking yourself what you need to instrument so that future you (and your team) can understand it.

⭐️ What does it do?
⭐️ Is it working as intended?
⭐️ Does anything else look "weird"?

Once you have put down your work and moved on, the original intent that is now in your head will be *lost for good*.

We have learned this about writing comments. We now lecture new engineers sternly about the importance of capturing intent in comments in their code.

We value this skill highly, despite the fact that comments can only tell you what you THOUGHT you were doing. You are an unreliable narrator!

Instrumentation is comments and documentation, times execution.

Instrumentation closes the loop of intention meets implementation.

If we simply made a practice of asking in every code review, "how will I know when this breaks?" and looked to instrumentation for our answer, our systems would be better-understood, transformatively so.

I know this sounds like a big, massive, HARD change, but it's not. With the honeycomb instrumentation beelines (soon you may be able to use them standalone too 🤞) it just means keeping an eye out for important identifying variables, and stuffing any new ones onto a blob.

Anyway. My intent isn't to shit all over product devs. Honestly, my first tweet was a detour from what I was going to say.

I just got back from my first product offsite, and it's just as insane that I could work in infra eng this long and never build a proper product 🙃

If I could redo the past decade+, high on my list would be giving a shit about user experience and product development.

Maybe learn what a persona was at some point *prior* to founding a company. 😁