Thread by @copyconstruct

Really enjoying this talk from the @FoundationDB summit on testing distributed systems. Makes a pretty compelling case for an AI-driven approach to testing. /thread

The talk identifies 3 main problems with testing - fragility ("your test comes to rely on properties of your system that are incidental - that are not the ones you thought you were testing"), lack of exhaustiveness and flakiness.

"tests are great for turning up regressions but almost completely useless for telling you about the unknown-unknowns".

All of the aforementioned problems are the symptoms. The real problem is that testing is still totally *manual*.

Even "automated testing" just involves Jenkins running your test suite manually authored by humans.

We need automated *creation* of tests, in addition to automated *execution* of tests.

That's the dream of autonomous testing.

Autonomous testing isn't a novel concept. It's being practiced in different forms already (fuzzing, property based tests, deterministic simulation etc).

On the FoundationDB team, there was a constant culture of taking testing problems and making them ... less "human-involved".

All of these autonomous testing techniques depend on your software being in the right *form* for them to test - you can think of the whole FDB deterministic simulation system as wrangling a big, hairy stateful distributed system into this form.

There were ACID invariants too complex to test with a simple assert statement. We had to design entire workloads dedicated to setting up the invariants and checking they were still true at the end.

The talk wraps up by listing some of the benefits we can reap from this vision for autonomous testing. It's a fascinating talk for certain. Do give it a watch if you have 30 mins - you won't be bored.