Five hours in @Reagan_Airport and still here; twice rebooked due to thunderstorms—hope I make it to Boston tonight for tomorrow's IEEE #reproducibility workshop.
As the @IEEEorg steps into the #reproducibility discussion, I'm really hoping they'll pay attention to terminology—"Terminologies for Reproducible Research" arxiv.org/abs/1802.03311
My assessment after reviewing literature from more than a dozen fields is that the predominant usage for #reproducibility is “same data+same methods=same results.”
The predominant usage for #replication is “new data and/or new methods in an independent study=same findings.”
The semantic distinction between "replicate" and "reproduce" goes back at least a dozen years, to "Reproducible epidemiologic research" by @rdpeng et al. doi.org/10.1093/aje/kw…
The Artifact Review and Badging effort of @TheOfficialACM, unfortunately, swapped the meaning of the terms "replicate" and "reproduce," as predominantly used acm.org/publications/p…
The ACM adopted definitions based on the International Vocabulary of Metrology, which establishes terminology for physical measurements. The IVM document contains no definition of “replicability” [PDF] bipm.org/utils/common/d…
The scenario for the IVM definitions is measurement of a physical quantity, and associated precision. Reproducibility of measurements involves changing at least one condition—e.g., the instrument, the location, or the operator—to measure the _same_ quantity.
Reproducibility of measurement is expressed by a dispersion statistic, like standard deviation—because it refers to the _same_ quantity being measured.
Along the way of creating the ACM terminology, based on the metrology document, some leap occurred. Arguably, conditions for physical measurements are a distant analogy for the complex processes of a full scientific workflow. #reproducibility
