My Authors
Read all threads
An example of reproducibility checking when data is restricted-access (or "reproducibility checking is hard, but never impossible").
I want to highlight a recent article in the @AEAjournals Economic Policy:
Leung, Pauline, and Christopher O'Leary. 2020. "Unemployment Insurance and Means-Tested Program Interactions: Evidence from Administrative Data." doi.org/10.1257/pol.20… Image
This manuscript was part of our earliest reproducibility checks - assigned to us in April 2019. It is an excellent example of a well-documented, and as it turns out, reproducible article. Even if the data is confidential and restricted-access! (1 Caveat: see the End)
The data underlying the analysis is administrative data from various state agencies in Michigan, and it would have been difficult and time-consuming for us replicators to request access (not that we don't sometimes try - stay tuned!) Image
We reached out to the authors and asked whether they would be able to add a 3rd-party to their research protocol - either a member of our team or somebody at their institution @UpjohnInstitute who was not involved in the original research.
The authors connected us with a research associate at their institution who had legal access to the data. Arms-length: the 3rd-party replicator followed only written instructions that are publicly available, as per our protocol aeadataeditor.github.io/aea-de-guidanc…
In fact, all information and code came from the researchers' replication archive at openICPSR, which is now available at doi.org/10.3886/E10970…. However, the replicator had access prior to publication of the archive, and what you see now is not what they had access to. Image
That's because the replicator identified and suggested multiple (small, but meaningful) edits to README and programs, to make the replication easier for subsequent replicators.
They also identified (minor) various numerical discrepancies, typically at the 3rd decimal place, that we tracked down to differences in the versions of the Stata package `rdrobust` sites.google.com/site/rdpackage…
After discussions with `rdrobust` authors and manuscript authors, the replication package now contains the exact version of `rdrobust` used for the analysis, even though a more current (and corrected) version exists.
Overall, the report received from the 3rd party replicator was useful confirmation that the code that anybody can see would work well on the data that few people can see, and materially improved the replication package.
Take-away message: While we cannot do reproducibility checks on restricted-access data in a timely fashion for all papers that use restricted-access data, there are still many opportunities to do so, and we will always attempt to do so.
In fact, since that paper was reproduced, we have been able to make similar attempts at reproduction for many other papers. We have signed NDA or DUA for various datasets, to obtain copies ourselves, or worked with others. In the past month alone, about 10 such papers.
Not all attempts to access restricted data succeed, but expect to see more such verifications in the future!
The Caveat: We were still refining our various checks and guidance documents. Authors did excellent job citing various data sources (in online appendix aeaweb.org/doi/10.1257/po…, but we did not ask for a data citation for the core confidential data. That would be different today.
Missing some Tweet in this thread? You can try to force a refresh.

Keep Current with AEA Data Editor

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!