a) logically impossible
b) a bottomless money pit
c) mostly junk (boring things being infinitely more common than interesting things)
d) ends up with a heavy bias towards errors, and
e) causes good people to lurch desperately towards bad ideas (like "AI")
just a few minutes ago i was having a conversation with @paulosman about the future of our profession as systems engineers, and he pulled up a quote about four ways to solve problems: solution, resolution, absolution, and dissolution. (e.g. squiretothegiants.com/2016/06/16/so-…)
logs are such a canonical example of a solution that teams are used to either absolving (ignoring it and hoping it will go away) or solving (an outcome that is good enough to move on).
if your problem is "do something with logs", a good enough answer is "pick any log vendor."
logs are a commodity now, a race to the bottom. you can ship them to any one of twenty or fifty vendors, pay a few cents per gig to not think about them, and sleep fine at night.
but what if the problem you are trying to solve is "understand my systems" or "understand my code?"
(er sorry, i got that wrong above ... "resolve" is the one that yields a good enough outcome. "solve" is do something that yields the best possible outcome. CRITICAL DISTINCTION ☺️)
if what you're trying to do is *understand your systems*, logs are a shit-poor way of trying to do that. we fall back to logs when that's all we have. but what if you didn't log the detail you needed to understand the problem in front of you?
what if what you logged is misleading, or flat out wrong? what if the problem only manifests when you examine system behaviors in aggregate, and is invisible from the perspective of any given host? what if it doesn't match up with the telemetry from your monitoring systems?
what if your logs are filled with useless spew left over from the last engineer who was frantically stuffing in shit to help her "step" through the code from the last outage?
worst of all, what if you don't know what to look for? logs _only work_ when you know what to look for.
the problem most people face is that they've been using logs for so long that they've wrestled them into a state where it's "good enough" -- everyone is familiar enough with the local whimsies that they know how to get what they need out to get thru most situations.
and so they completely lose sight of the fact that the problem they're trying to solve is, "understand my systems".
and they start shoveling more and more and more engineering energy into the gaping maw of their logging "solution". the problem instead becomes "managing logs".
many companies end up sinking so much engineering talent into this problem, you'd think they had a hybrid mission -- whatever their company does, plus log management.
logs for logs' sake will swiftly become a millstone around the neck of any team that lets this happen.
there's so much emotional attachment around this topic, it seems like many teams (and leaders) have come to imbue their logs with a kind of emotional security blanket magic.
(for which you can prob thank the millions of dollars spent on marketing by aforementioned logs vendors.)
but what is the problem you are trying to solve?
it is not "keep a record of every thing that happened".
it is not "search our logs".
it is not "keep everything".
the problem you are trying to solve is understanding your systems and the code you write that makes your business.
if the goal is to "solve" it (find the best possible outcome) and, in some cases, "dissolve" it (reframing the question or redesigning the system so it no longer exists) ... what would you do?
well, first identify the minimal set of logs that you actually do need (usually for compliance reasons). pick any log vendor, stash and forget.
then, of course, spin up a prometheus instance or a datadog account and -- LOLJK, you know what i'm going to say about observability ☺️
but if you CAN'T start using honeycomb or lightstep or similar, if you're stuck with what you've got, what can you do to dig yourself out of logs hell?
at LEAST you can shift away from random acts of logging violence towards emitting arbitrarily-wide structured data blobs,
one per service per request, containing the full context of the request, env, parameters and so forth. i've written about this extensively charity.wtf/2019/02/05/log…
aws apparently does this, and stripe calls it "canonical logs" stripe.com/blog/canonical… so it's not just my bullshit.
then ship these events to someplace where you can aggregate and slice and dice them. you can also tee them off to a honeycomb free account, just fyi ;) since that's basically what our client side integrations do.
btw, if you haven't yet done this refactoring and if you shudder to think how much labor it would take, check out @cribl_io. it's by ex splunk folks and it reconstitutes log spew into coherent events for you.
this is *not* a solution, but could be a bridge to your future. 😉
in the end, if your goal is to *understand your systems* then you need to be aggregating loads of rich context around the request, because that is what most closely tracks your users' experience.
this is observability.
logs can (*can*) be pressed into the service of this goal, if gathered and formatted and expressed in the correct ways, but the presence (or lack) of logs is orthogonal to the goal.
use logs or don't use logs, but don't let "logs" become the problem you solve for. the end.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
It felt, to me, like those participating were stepping very cautiously around a few of the third rails Jaana just tripped over. (💜)
"Work-life balance"
"Working hard vs working smart"
"Meritocracy"
The intersection of company tech cultures and expectations and performance.
These are hard, complicated topics, and there are some very good reasons for speaking carefully. People can pick up a sentence and run in the wrong direction with it, and do a lot of damage.
I have abandoned god only knows how many drafts on this topic, for that reason.
The question is, how can you interview and screen for engineers who care about the business and want to help build it, engineers who respect sales, marketing and other functions as their peers and equals?
It's a great question!! I have ideas, but would love to hear from others.
I said "question", but there are actually two: 1) how to hire engineers who are motivated by solving business problems and 2) aren't engineering supremacists.
Pro tip: any time you see someone confidently opining on what all good CTOs know or do, it is ✨bullshit✨
There is no stock template for CTO, or default set of expectations or responsibilities. It stands alone among the C-levels in that good ones are all over the freaking map.
This may not hold true for publicly traded companies. But in my experience, a good CTO can be:
* over all of R&D
* over engineering, like a VP eng
* like a principal eng or architect
* team lead for special projects
* a great senior programmer
(continued... 👉)
A CTO can also be:
* a great communicator and popularizer
* on the road as a devrel
* a field CTO, whose authority opens doors to big customers
* a product visionary who sweats the details
* more of a cofounder than technical contributor, sharing "company-running" duties w/CEO
Yeah, this is a fair caveat. If you're already a decent senior engineer and manager, it's kind of possible to split your attention between managing a small team and writing code.
But you aren't going to improve at either skill set. Those cycles get devoured by context switching.
Tech lead managers ("TLMs") are a mistake we make over and over in this industry. I've written about this a bit, but the definitive post was written by @Lethain.