Thread by @mipsytipsy

Sorry kid; you're fucked. Tracing is the single most cost effective investment you can make with that many services. Anyone you hire will just be cannon fodder.

That said, they are not wrong to be wary of the typical big bang deployment. Not for cost reasons, but whatever --

https://twitter.com/akasprzok/status/1207891294938374144

-- they're keying in on something real, so let's not sweat the rest. What you need to do is bootstrap a solution compelling enough to convert the unbelievers.

You need to:

✅ solve a problem that is currently intractable
✅ swiftly (an hour or two)
✅ cheaply (free? 🤞)

Fortunately, I can help. 🤗

First, go download this paper. (Btw, if anyone complains about the email gate I will take that as a request to block you so that all my future FREE ADVICE need not offend your precious lil eyes) info.honeycomb.io/guide-path-fro…

It's a super quick read, and one of the shortest, most practical things we have ever written on how to take the logs you've got and just...make them a little better. Bit by bit, inch by inch. No big bang heroics.

We ramble on a lot about the importance of structuring your logs. (Even got a new sticker 👇)

And I talk a lot about those arbitrarily wide structured data blobs, one per req per service, that enable you to achieve observability.

But this also -- guess what! -- unlocks traceability. Events are just spans are just events, with a unique id you persist and some fancy fields for ordering and visual hints.

Everything you do in this direction is an investment in your future. None of it will ever be a waste.

This is the best advice you're gonna find anywhere about how to construct your log events.

If you doubt me...we all found out just last week that this is exactly how AWS services log. It would be cool if I had known BEFORE spending a year figuring this out the hard way, BUT HEY

And pulling it all together with tracing.

"You don't have to do distributed tracing to start thinking in the tracing mindset."

Trust me. With 100+ services, you need to get on this. Might I suggest a full body tattoo?

So.. Coming back to your situation specifically.

What you want to do is take aim at one of the biggest, shittiest sources of pain. The service under active development, the thing waking people up st night. Any endpoint experiencing phantom latency. 💥Instrument that shit.💥

Don't start small. It's not worth your time. It's more work, not less, than attacking the most painful parts of the system, and the payoff won't convince anyone.

Instrument, ship it off to honeycomb, and start answering these previously intractable questions.

I'm stupidly confident this will work for you, because I remember doing it myself, and finding bug after bug after bug we could never see before.

And every fucking time I check the honeycomb user slack, someone else is having their own come to Jesus moment.

There's a ton of broken shit in your services; you are going to be horrified once you start rolling it out. That's okay. That's how you show value to leadership.

✅ Should take an hour or two.
✅ Use our free tier. Retention is low, but works for your purposes.

Many of you who are trying to convince your team or leadership to use honeycomb would do better to spend an hour or two simply solving a problem for your team. It's a lot easier than you may think.

Code wins arguments, as they say.

You do not have to go all in on Jaeger vs open tracing vs whatever, you do not need it on your roadmap. You do not have to spend a dime to start finding and fixing these mystery latencies.

You *can* bootstrap in a nimble way, starting with your hardest problems and working out.

And you can spare yourself and your team a lot of pain if you do it now.

🌈 good luck, let me know how it goes 🌷📈🥰