Thread by @MichelleNMeyer

This @jessesingal piece on the IRB aspects of Sokal 2.0 is quite good. That said, I do have a few additional thoughts. All cites, unless otherwise noted, are to the pre-2018 Common Rule, which was in effect at the time. (Are we still saying THREAD?)

https://twitter.com/jessesingal/status/1083792146543579138

First, the following distinction is critical:

1) What did PSU policy—which applies the federal Common Rule to all human subjects research (HSR) regardless of funding—require of Boghossian?

2) Is this good &/or consistent policy?

I’ll mostly address 1, w/a few comments re: 2

To begin with my bottom line, I agree with these sentiments from Singal’s piece.

As Singal explains, once you are subject to the Common Rule via federal funding or, in this case, via institutional policy (sites.google.com/a/pdx.edu/rese…), the first step in knowing whether you are subject to IRB review is whether your project meets the definition of “research”:

There have always been some edge cases here. The revised Common Rule (not applicable here) codifies certain activities as not-research, among them “Scholarly and journalistic activities...that focus directly on the specific individuals about whom the information is collected.”

Although this wasn’t codified in the old Common Rule (CR), IRBs have to apply the definition to edge cases, and they might have explored the possibility that this was journalism, not research. Two problems, though.

1st, if I may read the new CR into the ongoing definition of “research,” note the qualifying clause about focus on individuals. Sokal 2.0 didn’t investigate personalities; it investigated (if you like, exposed) alleged weaknesses in systems of knowledge production & peer review.

2nd, note that part of the def of “research” is that an activity can be BOTH research AND something else (QA/QI, demonstration project, journalism) & if so, that activity is treated as research under the Common Rule. See also hhs.gov/ohrp/regulatio…

This is also a problem for the interpretation that 2/3 of the Sokal 2.0 authors themselves offer for their project—namely, that it was not research but a “quality assurance investigation.”

1st, if QA/QI is ALSO research, sorry, you’re SOL. (Ask Peter Pronovost.) 2nd, there are reasons to doubt this was QA/QI at all, which usually is conducted by (or in collaboration with) insiders, local, & provides rapid, constructive feedback to the system humansubjects.stanford.edu/research/docum…

Whether Sokal 2.0 was “research” is 1 of 2 weak points in the case for required IRB review, but I lean towards Yes. It was “systematic” enough & investigation was designed to yield generalizable conclusions about “grievance studies” & hum journals, not just those in their sample.

In rejecting the claim that Sokal 2.0 was “systematic” & “generalizable,” I’ve seen some set WAY too high a bar. “Research” needn’t be an RCT, or replicable. Qualitative focus groups & interviews aren’t replicable but 100% count as “research” (though generally exempt—more below).

The fact that they *didn’t* publish in an academic journal, nor intended to, is irrelevant to whether the activity constituted “research” or not. As for claims no journal *would* publish their results—seriously? You need to read journals that publish a range of methods & quality.

Assuming an activity is “research” (even if it also something else), the next question is if “human subjects” were involved. The answer is yes. This isn’t a close question. Submitting hoax papers to learn how reviewers & editors behave was an intervention & they were the subjects

In Areo, 2/3 of the Sokal 2.0 suggest that anonymous reviewers are “exempted” from being subjects according to PSU policy (i.e., the CR). But exemptions apply to activities, not people. An activity can’t be exempt as to some subjects (reviewers) but not others (eds)—not a thing.

Singal, too, sometimes conflates non-human subjects research (an activity that is non-HSR b/c it is not-research &/or lacks human subjects) & exempt research (HSR that is nevertheless exempt from the Common Rule for policy reasons).

So let’s move on to exemptions, which is the next question to ask: is this “human subjects research” nevertheless “exempt”? Again, the clear answer is No. Submitting hoax papers constituted an “intervention” for CR purposes & no exemption then in effect applied to interventions.

Even the new exemption for “benign behavioral interventions,” had it been in effect, would not apply: it requires subject consent & no deception.

The other weak point in the IRB-review-required case I haven’t seen raised is that even non-exempt HSR generally only triggers IRB review if an institution committed to the Common Rule is “engaged in research.” This is an obscure but critical aspect of the Common Rule.

OHRP has guidance (which many unis adopt) & under it, I argued that—due to the limited role of the Cornell collaborators—Cornell was not “engaged” in the infamous Facebook mood contagion experiment, only FB was (where CR doesn’t apply). I discussed here wired.com/2014/06/everyt…

There, the Cornell researchers only helped design the study & write it up; they didn’t collect or analyze the data. In this case, apparently only Lindsay interacted with the editors, so there may be a similar case that PSU wasn’t “engaged.” Below: PSU & Cornell’s +explicit policy

A 2nd variant of this argument: perhaps Sokal 2.0 was moonlighting for Boghossian. Faculty occasionally have private lives & that some of us lead those private lives in evidence-based ways or pursue hobbies substantively related to our scholarship can’t plausibly trigger IRB rev.

E.g., suppose I conduct an N of 1 study of my diet. *If* my institution were “engaged” in research via my activity, it would indeed require IRB review/determination (yes, the CR covers self-experimentation thefacultylounge.org/2014/04/whose-…). But surely my own time/resources = no engagement

Was Boghossian moonlighting? I doubt it, but I don’t know and it’s an important consideration. I don’t see an explicit PSU policy about whether faculty may self-determine that PSU isn’t “engaged” in their extramural HSR, so he could be forgiven for thinking he didn’t need to ask.

Assuming PSU was engaged, though, PSU policy required him to ask them whether his project was non-HSR or exempt HSR (again, I say mayyyybe to the former, no way to the latter). See 4.1 & 4.4 here sites.google.com/a/pdx.edu/rese….

There’s a strong case that Sokal 2.0 was non-exempt HSR. So what would have happened had he sought IRB review? The 1st thing to note is—contra 2/3 of the Sokal 2.0 team make here—SUBJECTS’ CONSENT IS NOT ALWAYS REQUIRED, LEGALLY OR ETHICALLY, FOR HUMAN SUBJECTS RESEARCH.

For some reason, people on both sides of the “IRB wars” persist in spreading this fake news. A few minutes skimming the Common Rule (below) will disabuse you & I discuss more in Section II.B here papers.ssrn.com/sol3/papers.cf…

I would’ve approved (some version of) Sokal 2.0. Not only needn’t IRB review impede important audit studies, it can benefit the necessarily non-consenting subjects (yes, “subjects” in that case, not “participants”) by identifying & minimizing risks researchers may have neglected.

I am intimately familiar with the “IRB horror story” literature, have referenced it, & leveled my own criticisms of IRBs—especially when they presume to know how competent adults would balance research risks & expected benefits. See, e.g., papers.ssrn.com/sol3/papers.cf….

But when you are necessarily imposing risks/costs on subjects without their consent, and not for their own good, a fresh pair of eyes is more important, not less. I discussed some of this w/r/t the “chocolate-causes-weight-loss” sting thefacultylounge.org/2015/05/a-rema…

In this case, an IRB might have explored risk minimization possibilities like using a smaller N, using manuscript acceptance as a surrogate end point for publication, concealing editors’ and journals’ identities, and debriefing.

So although I am *quite* open to conversations about relaxing IRB jurisdiction over some research & investigator self-determinations, figuring out how to draft policies that do that w/o being under- or over-inclusive is harder than it seems, & I’d start w/consensual research.

Let me be clear: I assume, until proven otherwise, that Boghossian genuinely believed he was not required to submit to the IRB. The Twitter discussion of this has made clear how ignorant many—including the Sokal 2.0 team—are about IRBs/the CR, so this shouldn’t be shocking.

If so, the appropriate sanction is something like remedial CITI courses (punishment enough), not firing. I see no evidence that firing is seriously being contemplated for IRB noncompliance, but if things go down that path, hit me up & I’ll gladly weigh in on why that’s absurd/END