Thread by @chris_mcmahon

I kind of want to do a thread about build-test-deploy pipelines and why automated end-to-end tests are a critical part of the process. So...

First you have to write some code. You'll do whatever you do to make sure the code itself is sound. This is usually some combination of code review and unit tests. Your pipeline starts with running those unit tests.

You'll probably have to gather some components and make sure they interact correctly with some integration tests. Your pipeline continues with gathering these bits and running those middle tests.

Now you have some deployable code, but you don't really know if you have a working system. You could deploy to prod right now, but that would be irresponsible.

What you need now is what I call a shared test environment (STE). Your STE should be an avatar for production. The process that you use to deploy to the STE is the process that you will use to deploy to prod. Deploying to STE is the next bit of the pipeline.

An STE doesn't have to be persistent. It might be a docker image or a vagrant instance that anyone can stand up anytime and deploy to it. (My current minimum test environment is a 5-node Hadoop cluster. STEs come in all kinds of forms)

So your pipeline deploys to your avatar for prod, the STE. You still don't know if you have a working system though. This is where you need to have some end-to-end tests, "DOM to database" as a friend of mine likes to say.

This is where the haters step in and say "but Chris there are an infinite number of tests for even the simplest software system" and "but Chris only real humans can say if the system works".

This is where we need to put on the critical thinking hat and do some test design. First of all, only write end-to-end tests that work with data. Don't write tests for UI components, your unit tests did that already.

Don't write tests for data, your integration tests did that already. Don't test boundary conditions or mathematical operations, you've already covered all that earlier in the pipeline.

Your users have a finite number of things that they need to be able to do in your app. Write tests for those things. Make sure every test has at least one verb: type, click, select, enter. Make sure each test starts in one place and ends in another place, with a testable result.

Now you have proven that your users can do everything they need to do, you can pull the trigger and deploy to prod. Automatically if you want to.

But now the haters are going to say "but Chris no humans have looked at the STE, even if the tests pass there's no way to know if the system is OK" but they're wrong: we HAVE looked...

We had to design the end-to-end tests in the first place. We have already been deep in the system, doing exploratory testing, digging into the details of the functions-- we had to do that in order to write the tests at all!

In my experience of the last decade or so, creating and making good use of an STE is where orgs get stuck.

PS: when you have good, working tests for the finite number of things your users need to do, PLEASE stop writing new tests. End-to-end system tests are expensive. Don't make more of them than you need.