After 19 years of writing automated tests in a bunch of tech stacks I have developed an opinion or two on what makes a good test runner.
What would my ideal test runner look like? Here's a list of features.
🧵"Strap in", as the kids say.
What would my ideal test runner look like? Here's a list of features.
🧵"Strap in", as the kids say.
1) Tagging: an extensible way to annotate tests with metadata. This allows external tooling to implement features like quarantining (with expiration dates), marking a subset of tests as running before commit, and so on.
2) Pended/muted/ignored tests: Out-of-the-box support for marking a test or suite of tests as pending. Ideally this would just be a convention built on top of the general tagging system, rather than a special case.
3) lifecycle hooks: Let me define before and after hooks which run before specific tests, or specific suites or sub-suites. Support for both once-per-group and once per-test hooks.
4) Tag-aware lifecycle hooks: allow me to define before and after hooks which only run for tests with certain tags. Useful for things like setting up browsers, re-initializing databases, etc.
5) Modular configuration: some test runner configuration is global, some only applies to interactive test runs, some only to CI, etc. Allow me to keep those seperate, and specify which config to apply to a specific test run.
6) Extensible reporting API: Give me a reasonable set of report formats out of the box (JUnit XML, nice HTML, pretty colored console output), but most importantly make it easy to create custom reporters.
7) Allow multiple reports for a test run: I want a pretty HTML report, as well as JUnit XML. It is amazing to me that this isn't always possible out-of-the-box.
8) Test coverage: Make it really easy to generate test coverage using an external library, or just generate it natively.
9) Parallel testing: Run tests in parallel. Support opt-in and opt-out annotations for tests and suites. Support for disabling all parallelization. Support for uncoordinated parallelization (for CI).
10) Async testing: First-class support for async tests, don't expect tests to complete synchronously. Support for configurable timeouts, per-test, per-suite, and global.
11) Out-of-the-box support for "dev mode" and "CI mode": Dev mode means interactive UI, watch mode, only running tests for what changed, etc. CI mode means run the full suite, parallelized, with good report output. Ideally this is all powered by modular config, not a special case
12) CI server detection: The test runner should notice if there's a `CI` environment variable, and switch to CI mode by default.
13) Focused runs: Just run tests for one test, one test, one directory, etc.
14) Test detection: Most of the time, the test runner can find my tests without me having to tell it where they are. Support for specifying test paths, with globbing, etc.
15) Auto-load support files: The test runner should automatically load in support files (hook definitions, custom matchers, global helper functions, etc) based on a directory convention, with an option to override that convention with globs, etc.
16) File watcher: Automatically re-run tests when they change. Only re-run tests which have changed, or re-run those tests first, then the whole suite.
17) Semantic file watcher: automatically re-run tests when any production code that they exercise changes.
18) Dynamic/smart test ordering: Run fast tests first, run more relevant tests first, run tests that recently failed first.
19) Randomized test ordering: Run tests in a random order each time, but reporting the seed for each run so I can reproduce the ordering when needed.
20) OS-level notifications: pop up a notification when my test run completes. Configurable so I can only get notifications when a test fails, or when a failing test starts passing.
21) Debuggability: First-class support for launching a test in an interactive debugger. Bonus points if I have a super-easy way to go from a test failing in the CLI to an interactive debugging session for that specific test.
22) Deep IDE integration: Do whatever is necessary to make it really easy for an IDE to find out what tests are available, trigger test runs for specific tests or groups of tests and interpret test results.
23) Jump to test: A nice way to jump between tests and production code (I guess this isn't really a responsibility of a test runner, but it could maybe make it easier for an editor/IDE to support?)
24) Unix philosophy: A 3rd party expectation or matcher or test double library should be able to integrate with the test runner just as well as any built-in library. Exposing lifecycle hooks, ways to extend internal tests DSL, etc.
25) Arbitrary nesting of sub-suites: i.e. nested describe blocks in rspec-like test runners.
26) Nested contexts: Some way to define context that's relative to a sub-suite. i.e. the `let` feature in RSpec, or lexical scoping in JavaScript test suites.
27) Sensible null failures: Fail if a test suite is empty. (optionally) fail if a test contains no assertions.
28) Suite introspection: The test runner should allow me to generate a parsable "dry-run" report which describes the test suite structure (with metadata information), including which tests would have been skipped.
29) Duration reporting: Out-of-the-box information on how long each test took, which tests were slow, etc. This should be implemented using the public reporting API, rather than special-case.
30) Really good docs: Clear description of command-line options, plus in-depth discussion of fancier features like tagging, plus guides which show how to use the tool for specific scenarios.
I have additional features that are specific to a Javascript codebase, but I'll leave those for another day 😅
What did I miss? What am I dead wrong about?
