#Python tip: #hypothesis is good at finding bugs; however, often as not, the bug is in your understanding of what the code is supposed to do.
1/
1/
Initial belief: The JSON module is buggy because #hypothesis finds cases that don't round-trip.
Bugs in understanding:
* The JSON spec doesn't have NaNs
* A JSON module feature is that lists and tuples both serialize into arrays but can't be distinguished when deserialized.
2/
Bugs in understanding:
* The JSON spec doesn't have NaNs
* A JSON module feature is that lists and tuples both serialize into arrays but can't be distinguished when deserialized.
2/
Initial belief:The colorsys module is buggy because #hypothesis finds conversions that don't round-trip
Bugs in understanding:
* color gamuts are limited
* colors in one gamut may not be representable in another
* implementations are constrained by underlying specifications
3/
Bugs in understanding:
* color gamuts are limited
* colors in one gamut may not be representable in another
* implementations are constrained by underlying specifications
3/
Initial belief: A quadratic formula function is buggy because #hypothesis finds cases that don't round-trip.
Bugs in understanding:
* Floating point numbers differ from Real numbers: representation errors, round-off errors, catastrophic cancellation, overflow/underflow, ...
4/
Bugs in understanding:
* Floating point numbers differ from Real numbers: representation errors, round-off errors, catastrophic cancellation, overflow/underflow, ...
4/
So, the reason I like #hypothesis is that it falls in the same category as Alloy and TLA⁺.
They are tools that reveal flaws in your understanding or specifications of problems.
5/
They are tools that reveal flaws in your understanding or specifications of problems.
5/
Particularly for floating point code, by the time you've added enough constraints to satisfy #hypothesis, you've developed a pretty good understanding of what your code actually does.
And that is valuable even if the code under test didn't have any bugs from the outset.
6/
And that is valuable even if the code under test didn't have any bugs from the outset.
6/
Some color space specs are based on physical and/or analog processes.
The conversion matrices typically go out to three decimal places, well beyond what early color TVs could hope to implement exactly and beyond the ability of the human eye to differentiate the results.
The conversion matrices typically go out to three decimal places, well beyond what early color TVs could hope to implement exactly and beyond the ability of the human eye to differentiate the results.
That said, Alloy and TLA⁺ are specifically aimed at reasoning about specifications. That level of abstraction is both a strength and a weakness.
In contrast, #Hypothesis operates on real code and produces concrete failure cases.
In contrast, #Hypothesis operates on real code and produces concrete failure cases.
#Hypothesis has another super-power: test case reduction
Few tools attempt this because it is a hard problem and it doesn't have a perfect solution.
That said, it is nice to have when it works.
I like tools that get to the crux of a problem.
Few tools attempt this because it is a hard problem and it doesn't have a perfect solution.
That said, it is nice to have when it works.
I like tools that get to the crux of a problem.
