So here is a right, but wrong, attitude among programmers.
It's right in stressing the importance of testing. Historically, testing has been deficient when programming, especially open-source projects.
It's right in stressing the importance of testing. Historically, testing has been deficient when programming, especially open-source projects.
One of the side effects of the Rise of GitHub is improved testing. Among the things you get with GitHub has your source code repository is the ease of integrated tests and "continuous integration".
I'd like somebody to do a study of it, but it seems to me that open-source projects are more likely to have automated tests now, and those with automated tests have better code coverage.
Testing dramatically improves reliability and gets rid of bugs. It's really important. However, the above tweet is wrong: the sorts of flaws hackers find are ones that don't get taught by tests. You can have 100% code coverage and still have major flaws.
Buffer-overflows are things that would never happen by accident. Take the Blaster worm bug, for example. It happened when a component received a machine name longer than 16 characters. No testing would find the bug because no part of the code could generate longer names.
The flaw was that an internal interface between two components was made external, and an external hacker could create input that no existing code could generate, either by hand or the hacker writing a lot of code themselves.
I don't think fuzzing would've found the Blaster bug, either. MS-RPC packets are quite complex, with multiple length fields needing to change in unison. Randomly flipping bits would cause hostile packets to be rejected before the bug could be triggered.
The recent Exim bug is similar. The error is between components delivering a hostname that comes buried deeply within SSL packets. Exim already has thorough test suites, lack of testing is not the source of this flaw.
Testing improves reliability in the face of accidental failures, but not intentional failures. It's like your car, designed to be robust against accidental failures, but is powerless against somebody knifing your tires or cutting your brake lines.
