Thread by @ReformedTrader

1/ Is There a Replication Crisis in Finance? (Jensen, Kelly, Pedersen)

"Using a Bayesian model, most factors can be replicated, can be clustered into 13 themes, work in a new large data set, and are strengthened by the large number of observed factors."

papers.ssrn.com/sol3/papers.cf…

2/ NOTE: Stuart Ritchie has an interesting book about the replication crisis.
Science Fictions

https://twitter.com/ReformedTrader/status/1336398442403307521

Aaron Brown has an incisive review of Ritchie's book that is worth reading and then re-reading. This paper echoes some of his points.
amazon.com/Science-Fictio…

3/ Methodology differences from Hou:

longer sample
1-month holding (ignore 6 and 12 months)
weight caps (b/c of mega-caps)
15 additional factors
exclude factors not significant in original studies
CAPM α, not raw returns
Bayesian framework
Cluster similar factors before testing

4/ Factors are always constructed using the most recent accounting and price data ("old" versions ignored; "HML-Dev" principle)

Industry and analyst data factors excluded due to short samples

Factors without global data are excluded

L/S = high - low terciles, dollar-neutral

5/ "Factors demonstrate a high replication rate throughout the size distribution.

"10 of 13 themes are replicable with a rate of ≥75% (exceptions: seasonality, leverage, and size themes)

"Some alphas (e.g., for leverage) become significant after controlling for other factors."

6/ "OLS replication rates in developed and emerging markets are generally lower than in the U.S., and the frequentist Benjamini-Yekutieli correction has an especially large negative impact. This is a case in which the Bayesian approach to multiple testing is especially powerful."

7/ "The close alignment with the 45° line demonstrates the close similarity of alpha magnitudes in the U.S. and ex-U.S. samples, but shorter international samples widen confidence intervals. This is the primary driver of the drop in OLS replication rates outside the U.S."

8/ "While we can rule out pure alpha-hacking (or p-hacking), there is some evidence (from the dots' concave shape) that the highest in-sample alphas may either be data-mined or arbitraged down."

9/ "Imagine a Bayesian observing new factor data
in real-time and updating her beliefs.

"When beliefs are confirmed by additional data, the posterior mean does not change. Nevertheless, we learn something because our conviction increases as the posterior variance is reduced."

10/ "It is only in marginal cases where BY and EB disagree.

"We find that the out-of-sample performance of factors discovered by EB but not BY is positive on average and highly significant. This suggests that the BY decision rule is too conservative."

11/ "The stable replication rate in Figure 9 suggests that the replication rate among the observed factors would be similar even if we had observed the unobserved factors."

12/ "World factor alphas tend to be economically large, often above 0.3% per month, and tend to be highly significant in most clusters. The exception is the leverage cluster, where we also saw a low replication rate in preceding analyses."

13/ "The ordering and magnitude of alphas is broadly similar across size groups. The Spearman rank correlation of alphas for mega caps versus micro caps is 73%.

"U.S. factor alphas share a 71% (48%) Spearman correlation with the developed ex.-US (emerging markets) sample."

14/ "Value factors become stronger when controlling for other effects because of their hedging benefits for momentum, quality, and leverage. Surprisingly, the leverage cluster becomes one of the most heavily weighted, in large part as a hedge for value and low-risk factors."

15/ "The code, data, and meticulous documentation for our analysis are available online."
...
bryankellyacademic.org

and
github.com/bkelly-lab/Glo…
...

16/ Related research:

Factor Performance 2010-2019: A Lost Decade?

https://twitter.com/ReformedTrader/status/1245559396714729473

Resurrecting the Value Premium

https://twitter.com/ReformedTrader/status/1315820031729496064

Global Factor Premiums

https://twitter.com/ReformedTrader/status/1198106865315405830

Do Factor Premia Vary Over Time? A Century of Evidence

https://twitter.com/ReformedTrader/status/1183099984285073408

17/ "If you measure value 100 different ways and average them, you are not testing 100 independent factors as is, unfortunately, sometimes asserted. You are not cherry picking the best way but creating, in our view, a more robust factor."

https://twitter.com/CliffordAsness/status/1374730489328762888