Thread by @Kunkakom

Fantastic work (@The_JOP) focusing on political representation with value for the entire discipline.
Exemplary of good science, I think.

▶️Multiverse analyses to assess robustness of earlier findings to model specifications
▶️Collaboration between replicator & original authors

https://twitter.com/JoeySaraceno/status/1317215137648975872

After Hansen&Treul reported evidence for a substantive finding in a JOP article, Saraceno reproduced the analysis and found inconsistent evidence for different model specifications.
The journal then invited all three authors to team up and work together to check the robustness.

The newly minted team decided that multiple analytical choices are reasonable for six variables, leading in combination to 288 possible models with reasonable analytical specifications.

Running all these models shows significant p-values only in 11% of cases. (The authors also show the size of the coefficients.)

Based on these findings, the new team of original authors and replicators agree "that we should no longer conclude that there is a significant association between the size of the LGB population in a district and the bill sponsorship activity of its representative."😲

Obviously, this strategy is relevant far beyond the study on political representation.
And what I great example of good science this is:

Great to see that someone undertook the work of reproducing an existing study to check its robustness
Great to see the openness of the original authors to check and overturn their earlier conclusion
Great to see a general interest journal initiate all this

A few comments:
Using multiverse analyses to overturn earlier results is spectacular. But that's not their only value.
Instead of showing that we should believe not-X instead of X, sometimes they indicate that we just cannot be so sure of what we know. Which is also valuable.

Also, multiverse analyses may not simply be used to check findings from published studies of other authors but they should and, I believe, will become commonplace to assess the robustness of many of our own findings that we will publish in the future.

Some studies already do this. E.g., Metaketa-@EGAPTweets in analyzing their field experiment. In this vein, it is not entirely correct to state that "this article introduces political scientists to multiverse analysis through an application".
advances.sciencemag.org/content/5/7/ea…

A good alternative to multiverse analyses are specification curves which may provide more informational value, particularly when testing thousands of models with many varying variables.
philippmasur.de/2020/01/02/how…
github.com/hhsievertsen/s…

These tools create huge amounts of information. A great tool is Boba which allows readers to navigate all these models themselves in an interactive appendix.
You can see yourself which variables make a difference

https://twitter.com/Kunkakom/status/1284415465314570242

github.com/uwdata/boba