Thread by @marketsensei

@LauraALibby raises an important point and question (thanks for the kind words Laura).

Maybe the high FDR we find is because the experiments have too small sample sizes and low statistical power?
>>

https://twitter.com/LauraALibby/status/1477491345707003904

Same question was also asked by a reviewer. This is where peer review improves a paper, IMO.

So we did two types of analyses (in section 5.3):
1. We estimated what the power is in these experiments (spoiler: not so low).
2. We asked what the FDR would be with 100% power.
>>

For the effective power in the experiments, the table below shows that it is 50-80% depending on significance level used.

50% sounds low, but the following analysis shows you can't improve much on FDR.
>>

In the analysis we plugged-in 100% power into the FDR formula.

This is a theoretical, unachievable best case scenario.

Why unachievable? Because achieving 100% power requires rejecting every hypothesis regardless of the result, which clearly will increase false positives.
>>

The chart below (Figure 4 in the paper) shows the estimated FDR is in the data (green line) vs. the best case scenario with 100% power (red line called minFDR).

We can see that increasing power to even 100% helps somewhat, but not dramatically.
(Fin)

@lizzieredford hope this thread also answers a few of the points you raised.