This story (from news.ycombinator.com) is a great example of tracking down "impossible" bugs in code. I mention this because it's the sort of debugging effort they never tell you about in school.
marcan.st/2017/12/debugg…
marcan.st/2017/12/debugg…
As a programmer, you'll eventually encounter a bug that's "impossible", that you can't figure out no matter what you do. It's a wall that you hit, and it's very demotivating. Programming school doesn't warn you that this will happen, or teach you how to cope with it.
Big engineering organizations are have two groups of programmers, those that get stumped by bugs who must ask others to help them figure out what's wrong, and programmers skilled at figuring out bugs, helping other people out.
I think the biggest difference is self-confidence. When you believe you can't solve a bug, then you can't. When you've solved many impossible-to-solve bugs, you believe that you can, and will keep working on the problem until it's fixed.
When you look backwards at an impossible-to-solve, you wonder about the impossibility of it. It's just an obscure thing, one of million possible things, how could anybody have tracked it down? The above example is a great demonstration: small stack size on VDSO.
The above story explains how to find 1-in-a-million things: bisection. First, reliably and quickly reproduce the bug. Second, divide things in half, and figure out which half has the bug.
Do this 20 times, and you'll find the 1-in-a-million obscure bug. It doesn't matter that you've entered obscure territory. The above story makes it seem like the programmer already knew all about the VDSO. But maybe he didn't.
He continually split the problem in half until he figured out the difference was in the module named "VDSO". At this point, he could research all about the VDSO, going from "never having heard of it" to "the expert teaching coworkers about it".
When you explain your impossible-to-solve bug to your coworkers, they'll be impressed by how much you know about an obscure corner of tech, and how you could possibly have found such an obscure thing. But you started just as ignorant of the problem as them.
Confident programmers know they can solve impossible-to-solve bugs not because they know all the technical bits, but because they know they can learn the necessary bits as they go along.
Confident programmers know that they find a needle-in-a-haystack, not by finding the needle, but by getting rid of the hay until only the needle is left. By increasing the area where the bug isn't, they decrease their search until only the bug is left.
Anyway, the moral of these tweets is that you've got this, that what the guy did to find that bug wasn't as impossible for you as you think. All impossible to find bugs are actually quite possible.
The only major thing there was reproducibility. He could reproduce the bug that others failed to reproduce. Once you have that, you have the solution, and nothing will stop you find from finding the solution.
