@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi There is some very weird stuff going on in this sample (as pointed out in the open letter), and given the high response rate to an online survey, lots of hard questions need to be asked of CINT (the survey company) about how they get respondents.
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi I worry about the data source more than the analysis (though people certainly can make coding/labeling errors, and data/code repositories and code review are helpful). I can imagine a lower rate of respondents >60yo in an online survey. And maybe low rate of GERD depending...
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi On how the question is asked. Asking about "heartburn" tends to be a lot more sensitive than asking about GERD (jargon). But ~16x lower vs a comparable sample seems too much. The gender ratio is odd, but odd things happen. But when you get to the high RR for SARS-CoV-2 in 30-39..
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi This gets pretty concerning. We have gone beyond unrepresentative to completely weird. Then the education/LatinX/income/SARS-Cov-2 categories creating a cell that seems very, very unlikely makes me worry about miscoding vs. a data source problem. I wonder how CINT recruits people
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi Is it possible that there is a financial incentive for respondents to claim that they are high-income/low education/LatinX people (does CINT pay more for some demographic groups?) Trying to imagine worst-case scenarios that could cause this kind of weirdness...
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi I don't have the answers here, but there are lots of questions. And it is the kind of situation that makes me wonder if we would be better off as a field with an ecosystem of preprints, transparent data & code repositories, and code review before publication.
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi Looked at one of the many CINT survey sites (Crowdology). The user reviews are interesting. One could imagine that Jeff could try a few online personas and figure out that Latinas with a HS education and >200K income rarely get screened out. Worrisome. 
@MaartenvSmeden @AllonKahn @georgemsavva @IBDMD @ADAlthousePhD @doctorira @tmorris_mrc @BrennanSpiegel @NYUDocs @statsepi I have no idea whether this could have actually happened, but I worry about worst case scenarios and incentives. I once ruined the lab sequencing gels for a week by washing the plates with a pump soap with moisturizer in it (stick with Alconox, people). Weird stuff happens.
