Thread by @danluu

Is there anyone who's writing about different problem solving approaches / styles? An example of the kind of thing I mean (but, incomplete, because it would be nice to see more than two approaches to a problem and I'm only going to discuss two for this example):

Once, at a meetup Matt Singer was hosting, Brendan Gregg asked me what I was working on, and I mentioned that I'd recently written a little (5kLOC) parser to parse every line of every dmesg we had in our datacenters to audit machine health issues.

Of course, Brendan had done a vaguely analogous thing for Netflix and he showed me what he'd done, which was so much in his style that I think that if you saw the result without knowing who did it, you'd say "wow, this looks like something Brendan Gregg would make".

It had a cool visualization where some of the power of the visualization comes from utilizing your brain's pattern recognition ability.

And it was interactive, you could click to get a lot more info on what was going on (and maybe you could mouse over things? I forget).

My solution was basically the opposite. I don't think it's better or worse (there are pros and cons to each), but I think it was the better solution for me and I suspect I would've failed if I'd tried to produce something that was good in the ways Brendan's solution was good.

My solution used one of my standard approaches at work: define a metric (in this case, higher was worse) and then show that the metric being higher is strongly and causally related to severe problems (data corruption, order of magnitude increase in tail latency, incidents, etc.)

And, for each counterargument people would have (which I already knew because I'd informally talked to people about it), have a rigorous explanation for why the counterargument is wrong, e.g., one I knew was going to get was that a machine going down is fine because

we sometimes lose TORs without incident, which is equivalent to losing an entire rack, so it shouldn't be bad to lose a host.

But I analyzed significant load-related incidents and found that a single host going down was implicated in a double digit percentage of those incidents,

so we clearly were not operating all services in a way where it was ok to randomly lose hosts.

All this stuff went into a document. The causal link and the counter-counter arguments were the majority of the document, by volume, but also all at the end, so people can skip it.

The other big chunk of the doc was a breakdown in various dimensions (host age, SKU, cluster owner, whether or not the host used the host health management system built by the aurora/mesos team, etc.), to determine the causes of the problem and what mitigations would be effective

And then, at the top of the document was a summary, X% of hosts have some critical health issue, with a description of what could be done about it.

I'm not good at making interactive visualizations the way Brendan is and his approach would've been very hard for me.

But I'm ok at finding a data-driven description of a problem in a way that knocks down all the objections one might use to avoid addressing the problem, so that's what I did.

I'm sure there are other approaches that would work and I'd love to see more discussion of approaches.