Thread by @emilymbender

Some reflections on media coverage of tech/science/research. It seems to me that there are broadly speaking two separate paths of origin for these stories: In one, the journalist sees something that they think the public should be informed of, and digs into the scholarship.

>>

In the other, the researchers have something they want to draw the world's attention to. But there are two subcases here:

Researchers (usually in academia) who see a need for the public to be informed, either acutely (ppl need this info NOW) or long-term (science literacy).
>>

Subcase 2b: PR orgs (usually in industry) want the public to know about their cool research, because it serves as positive marketing for a company, halo effect of competence, etc etc.

>>

(Universities have PR orgs, too, which I think serve the dual purpose of helping scholars reach the public in cases of 2a above and promoting the university --- but here what's being sold is the university as a place of scholarship...)

>>

I'm seeing what I think is a pattern of tech co PR orgs + journalists together making news out of "new paper!!!1!" where the paper has not been peer reviewed. Sometimes, I get asked to comment on these papers, with really fast turn around.

>>

The typical pattern is that the paper is "embargoed" so that only some journalists and in some cases people they ask for comment, can see it.

>>

In the cases I've seen, the papers have been preprints (think about to go up on arXiv) and not peer-reviewed. Pushing out PR like this thus feels like circumventing the peer-review process while claiming to do scientific research.

>>

I'd like to see the tech media refuse to bite on that bait, when Meta/Google/Amazon/whoever says: we'll give you early access to this paper before we "publish" it tomorrow. If it hasn't been peer reviewed, getting a couple of other researchers to skim & comment is no substitute.>

That's all just contributing to the #AIhype cycle, it's not informing the public, and it's not like there's something there that can't wait for peer review.

>>

I can imagine folks saying: It's better that these big tech cos put the research out rather than keeping it behind closed doors. But I'm really not convinced, for at least two reasons:

>>

1. With all the big AI lab papers flooding arXiv (and our conferences), we still aren't getting transparency on the really key issues, like how content moderation is actually done, what's actually in the training data, how users are targeted & user data monetized.

>>

(Never forget that Google wanted to suppress the Stochastic Parrots paper, which wasn't even talking about anything specific being done AT GOOGLE.)

>>

2. Big tech attracts researchers with money, sure, but also with prestige. If researchers in the big tech labs don't get to publish, there will be less incentive to go there. But that is fully compatible with peer review, rather than publication by press release. /fin (for now)