Thread by @simonw

I wrote about documentation unit tests: a technique I've been exploring recently to use automated tests to ensure documentation exists for the key concepts contained within a codebase: simonwillison.net/2018/Jul/28/do…

The key idea is to use introspection to figure out which concepts exist within the code (for Datasette I'm listing plugin hooks, view classes and config options) and then use regular expressions to check for matching headings or labels within the docs

They're also a fun way of exploring some of the advanced features of @pytestdotorg - in my article I demonstrate pytest fixtures,
parametrized tests and how the xfail decorator can be used to ship the tests without breaking the build even when some docs have not been added yet

They can also be implemented in a surprisingly small amount of code: here's my implementation of a test ensuring every Datasette plugin hook (powered by pluggy) has a corresponding section in the documentation
github.com/simonw/dataset…

I'm experimenting with cross-posting to Medium, so this piece is also available at medium.com/@simonw/docume… - I guess that means I should ask for claps there? I'm trying to figure out how Medium distribution works.

Here's what a failing test run looks like - the test_view_classes_are_documented() test runs once for each introspected Python class, failing for the one which isn't mentioned in the labels extracted from the docs by a regular expression