Thread by @mipsytipsy

finally got around to reading this piece that was going around last week. it's a great piece! although the title is actively misleading. i guess it's cool that "observability" is now considered title clickbait? 🤷‍♀️ medium.com/windmill-engin…

the throbbing heart of this piece is all about why testing and developing distributed systems is hard, in full painful glory. and ho boy is it hard.

he complains about the awkwardness of developing using prod tools. which .. sure, but that's not the key point here.

the critical point here is not primarily that o11y is cumbersome for development, it's that o11y and a prod-like dev environment is now the ONLY OPTION that delivers reasonably trustworthy results. in other words,

~ make dev like prod, then
~ test in prod.

this is the direction that all of software development is trending in. once upon a time you could just stuff your monolith and db in a container each and call it a day. now you have too many distributed components to trust those results.

and yes! this is a PITA to manage!

the whole point of the piece is to serve up this demand for a "specialized local development tool", like the one the author coincidentally is working on.

which i was all prepared to argue with, because ✨no!✨, we need more tools that ✨SPAN DEV AND PROD✨

but liz assures me that is in fact what Tilt is designed to do. so i will hold my fire and express my delight if so. ☺️

it is definitely time for more local development tools to span and smooth out the boundaries between pre/post prod.

ah -- one last thing, the piece details at length the pain of jumping from datadog to honeycomb to lightstep.

i agree, but this is self-inflicted. you can do all three using just honeycomb. why pay to store three times, why copy paste between tools? makes no sense.

you can set up a high level trigger to alert you if a threshold is reached using honeycomb; click the graph to dive deep using honeycomb; click any event to trace using honeycomb; click the span to zoom back out to events using honeycomb. #thereifixedit #notobservabilitysfault

quibbles around naming and tooling choices aside, this was a great piece about the problems with development and testing in the age of distributed systems. (just leave poor o11y out of it)

the solution, we all agree, is to: ✨Make Prod Boring✨ medium.com/windmill-engin… ☺️