Profile picture
Hillel @Hillelogram
, 17 tweets, 4 min read Read on Twitter
So on the sorta-not-really recommendation of @rtfeldman I watched "Integrated Tests are a Scam" () and I have opiNIOns

First some things I like: thinking about decomposing tests into subtests. Testing vs checking. Formalizing your mock objects.

Rest: =(
Okay, so speaker's core thesis is that you shouldn't use "integrated tests", which he defines as, roughly, testing a cluster of things where, if it fails, you don't know where the problem is. It encompasses integration tests, E2E tests, etc. Compare unit- "isolated"- tests.
(I have a different, more in-favour definition: integrated tests are tests where a possible failure mode is "emergence", bugs that only appear when multiple things interact. Emergence is a common source of incredibly dangerous bugs. 'Isolated' tests can't help here.)
Claim one: Apparently, though, "integrated tests" don't put design pressure on your code. This means you can be sloppy.

This assumes that

1. All testing must put "design pressure" on your code, and
2. The design pressure from testing is sufficient to get good design.
If (1) isn't true, then it's fine to have integrated tests. If (2) isn't true, then "integrated tests don't guarantee good code" is a red herring, because isolated tests don't guarantee it either. You still need good specific design effort!
(Tangent: in general, the reason I'm skeptical of the "TDD is about design!!!" crowd is that a lot of -- not all, but a lot of -- advocates I've met say you don't need upfront design because you have TDD, and you don't need sophisticated testing because you have TDD. Goalposts.)
To support claim one he gave an example of a workshop on a "connect five" game, where the attendees had to make a small change. "Just a 5 into 6!" And because the code was so bad, it was impossible to make the change.

So integrated tests are bad, QED.
Okay, two problems: one is that he never said how it was tested! For all we know it could have been smothered in unit tests and still had bad design. We have to take it on faith!

The BIG problem: the "small" change? Having different win conditions for the human and the computer!
Claim two: there's lots of possible paths in integrated code. So you have a combinatorial number of tests to write!

Agreed, but that's because the system can do a lot of things! That's why we use fuzzers and PBT and stuff to help generate tests for us. Systems are hard!
His solution instead: let's write contract and collaboration tests! Replace each side of the module boundary with a mock, and then write tests that verify the mock is behaving like the actual object does. Then your "integrated test" is a series of isolated contractboration tests.
I'm not doing the best job summarizing it, so I'd recommend watching from about 25:00 to 40:00 if you want a better description.

It's a neat idea! But I have two big objections:

1. His "contract tests" could be much MUCH better
2. This doesn't help at all with emergence.
I'm a huge fan of contracts: statements about code behavior as part of the code itself. Violating a contract is a code error. Usually done (quite effectively) as runtime assertions.

def list_pop(l)
require len(l) > 0
ensure len(l'new) = len(l'old) - 1
ensure out == l'old[0]
But the speaker wants them encoded as individual tests, ensuring that no other unit tests, runtime behavior, integrated tests, or dev testing can benefit from them.

Which is a problem, because code contracts REALLY nicely solve his "integrated tests" isolation problem.
When you embed contracts in your code and run an integrated test, if the error is because of a bug 15 functions deep... that function's contracts fail, and the error is localized.

I talk more about this here: hillelwayne.com/talks/beyond-u…
And this is important, because emergence isn't something you can isolate. His solution is to replace integrated tests with more isolated tests, via mocks. This is faster but assumes that integrated tests are there because of bigness, not systemic issues.
I think the difference in mentality is clearest when he says "if it passes the tests, it works." He then defines "works" as "we can't think of anything else to test." But the whole point of emergence is that we _can't_ think of it: it arises from the complexity of the system.
Summary: he finds a couple of pain points with "integrated tests" but misidentifies them as the root problem, not a symptom of the problems of emergence. His solution doesn't address emergence whatsoever. A similar technique (code contracts) does help deal with it somewhat.
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Hillel
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!