Thread by @dsquintana

Including a power contour plot in methods sections of papers would drastically improve the interpretation of results.

Here's why...

Let's say you designed your study and paired-samples t-test to reliably detect an effect size δ = 0.3.

Maybe that's the minimally interesting effect size? Maybe that's all you can afford? That's beyond the point for now, but check out @lakens on this daniellakens.blogspot.com/2020/08/feasib…

Here's the power contour plot for this scenario ⬇️

If the reader thinks that interesting effect sizes are LOWER than δ = 0.3, then it's easy to see that the chances of reliably detecting such effects drops pretty quickly.

But let's say a study was designed to reliably detect effects δ ≥ 0.8. In some areas, like psychophysics, true effect sizes of δ≥ 0.8 are plausible. But in most other areas of psych this is quite large and unrealistic.

Let's have a look at this power contour plot ⬇️

Imagine this was presented with a paper in a field where true effect sizes of δ≥ 0.8 are unlikely. It's very easy to see that you can't reliably detect more realistic effect sizes and that larger sample sizes would have decreased the effect size that could be reliably detected

In this scenario, if the true effect size is δ ≤ 0.54, then there's less than a 50% chance that this design is likely to detect this effect size.

In other words, it's more likely to miss this effect than detect it

With a power contour plot in the methods section, readers (and reviewers) can quickly see what sort of effect sizes can be reliably detected and then make up their own mind whether these sort of effects are realistic and whether important effects sizes cannot be reliably detected

Sure, people can just write down the effect size that can be reliably detected with the study design. This is better than nothing, but people have different conceptions of what's considered "important".

Power contour plots show you a full range of effects, at a glance