In strategy development, false positives (Type-I) can cause losses but false negatives (Type-II) can cause missed profits. This problem cannot be solved quantitatively at a high level. The answer is experience and no algo has that (except if it's a sophisticated expert system.)
A false positive is when you select a strategy that was the result of data snooping or over-optimization but the Sharpe, t-stat or p-value looked good. These may fail immediately but in some cases can remain profitable for many years due to luck of persisting market regimes.
A false negative is a missed discovery because among other things someone used a procedure for adjusting Sharpe, p-values of t-stats that ruled out the strategy as one generating returns from distribution with zero mean. Again, a regime change could make the strategy profitable.
Never forget that all statistical tests are conditioned on past data. There is no test about the future. We reject/accept strategies based on historical performance but the future may turn out to be somewhat different than the past.
Regime changes are unforgiving especially to market statisticians. Since this is an unsolvable problem, a meta-strategy for developing/using strategies is necessary but not sufficient. Eventually those who rely just on the past face absorbing barriers.
I wrote a paper in 2016 with some ideas and examples. It's a simple paper about the limitations of quantitative claims about strategy development. papers.ssrn.com/sol3/papers.cf…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
