Thread by @isocrime

I’m still thinking about this thing my undergrad mentee said to me ages ago about how come journal club is so hostile and critical, and shouldn’t we strive to create a more positive environment

and I don’t think we should, but it was hard to articulate why

for background, Academic Journal Club is when a dozen-ish scientists take a look at some freshly released research and argue about if it’s any good or not.

generally this involves a lot of nitpicking and suspicion.

science runs almost entirely on trust. it’s laughably easy to fabricate studies, and there’s big incentives (clout, $, mainly $)

independent replication is great but sometimes a result took 5 years to get and there’s 2 people in the world who can make the technique work…

so naturally, as we have no choice but to believe our fellows are honest, scientists are all skeptical bastards

plus, even with perfect ethics, science is HARD. sometimes the best analysis you can do just ends up being…wrong.

i remember an article breathlessly declaring that 75% of research can’t be replicated and therefore science is DOOMED

and as a biologist I was like “yeah dude we have already taken that into account, I barely believe 25% of the shit I did with my own two hands”

which brings us back to journal club. the reason it’s so harsh is because we’re doing our very, very best to use all of our combined expertise to pick out the good 25%.

because if you base your research on previous work that turns out to be bullshit, that is a big hassle

we also use journal club to teach; reading papers critically is both how you learn the known facts in your field and how you build essential bullshit-detecting skills

on the leading edge of science there’s no textbooks; it’s all squabbling over beers about dr liu’s new stuff

(and yes, sometimes you have beef with dr liu because she disagrees with your lab’s pet theory about stem cells and perhaps there is some trash talk)

my mentee critiqued journal club from a sort of social-justice-y perspective — isn’t this going to drive away students who aren’t aggressive, loud, competitive, and favor confident white dudes who are more likely to have those traits

wouldn’t it be more inclusive to be gentle?

which is a valid critique; not everyone is going to learn well in the midst of “dude actually if you look at figure 3C they didn’t include a scale bar and that means they’re trying to trick us”

and yet…every scientist MUST learn to be critical. we MUST ask “what if it’s wrong”

“what if a person I like is wrong”

“what if someone in power is wrong”

“what if /I’m/ wrong”

those aren’t comfortable questions. ten scientists in a room aren’t going to agree (ever) about the answers and boy howdy do we all have emotions about our positions

i don’t know if it’s possible to make this kind of exercise conflict-free.

maybe I want to be making the case for “I respectfully disagree”

Sarah Schumann has a book called “conflict is not abuse” that I think of often arsenalpulp.com/Books/C/Confli…

journal club, for all of its cantankerous nature (and despite my tendency to read the paper being discussed as it’s being discussed rather than ahead of time), taught me that respecting someone’s ideas doesn’t mean taking them at face value

I think it’s important to be able to look at something, say “I care enough to figure out if I think this is true or not,” and have that process of attacking an idea from all sides come out as a net positive, whether it holds up to the scrutiny or not

postscript: questions I ask when I read basically anything after 9 years of journal club

- does the author have motivation for bias
- what’s the author’s expertise, do they have enough to make these claims
- is this sample biased
- are you selling something
- who is the audience

- do I know enough to evaluate this
- is there a possible alternate explanation
- will this still be accurate if it’s repeated at scale
- what was that survey design like
- when was this written
- what do experts think about this author
- has it been repeated