I've been spend a bunch of my "spare" "manager" time lately working on fixing @EnvoyProxy test flakes. A few thoughts on testing. 🧵
We have invested heavily in Envoy testing. Our line coverage is > 97% split across unit tests (with mocks), integration tests (fake clients and upstream servers using real networking), and fuzzing. The effort here pays off: our velocity is high with relatively few regressions.
While all 3 types (unit/integration/fuzz) provide value, integration are the biggest bang for the buck from a functional perspective. They cover real end-to-end scenarios and provide high confidence that a feature actually works.
However, by their very nature, integration tests are highly concurrent as they rely on multiple threads for the clients, server, and upstream hosts. Furthermore, our CI runs across multiple platforms (Linux/OSX/Windows) and architectures (x86/ARM) with variable speed.
Because of the variability and project growth, we have seen an increasing number of flakes in the integration tests. Interestingly, this is not all from slow CI. ARM CI actually runs *faster* than x86 and we have seen new flakes from the speedup.
Fixing the flakes is not easy due to the inherent concurrency and black-box nature of the tests. I have come to realize that testing Envoy thoroughly and without flakes is actually a lot more complicated than Envoy itself!
The other thing I have come to realize is how much of a productivity loss flaky tests can yield in a large SW project: the entire enterprise slows down as engineers fight with CI. Thus, de-flaking tests is one of the most beneficial things a project can invest in.
The moral of the story: thorough test coverage is critical for maintaining velocity in a large project, but flaky tests push back on the productivity gain. Invest in your tests, and work on making them flake free!
