Thread by @NoahHaber

Milojević: Across the board, the days of the single author paper are increasingly behind us. Science teams are getting bigger.

It's not necessarily that bigger teams are more productive, but that more productive teams get bigger.

#metascience2019

Milojević: While most fields have greater interdisciplanarity with bigger teams, that's not true for highly specialized fields, like high energy physics.

#metascience2019

Side conversation from lunch question w/ @SydWood13:

Things have been pretty "tame" at #metascience2019. Not a whole lot of of (public, anyway) controversy/argument/fighting. Seems weird, given that we're a critical argument-prone bunch.

What fights do you want to see?

@SydWood13 Milojević: These were a lot of things that in my mind are all connected.

Noah: It is, indeed, all connected. Almost everything in metascience is connected to everything else in critical ways, and the high level sprinting tour was great!

#metascience2019

@SydWood13 Dorothy Bishop on cognitive biases in science in statistics (YEAH!!!). General public seem to hate selective reporting, nearly as much as they hate outright fraud. P-hacking ~= paltering (saying something technically true in a misleading way)

#metascience2019

@SydWood13 Bishop: I literally "forgot" about an article to cite that didn't agree with my theory; a paper by my PhD student.

It isn't necessarily done intentionally. We just do this as part of our wiring.

@SydWood13 Bishop: What don't people paltering in science is so bad?

"Can be unintentional - hard to establish blame
Need to 'tell a good store' in limited space
Social pressures ('everybody is doing it')

#metascience2019

@SydWood13 Bishop's talk is reminding me of an interesting argument (which I am sympathetic to) that we should get rid of the literature review in the introduction, as it is a poor filter of quality and promotes entrenching false/weak viewpoints.

#metascience2019

Jevin West up w/ how to deal with echo chambers in science! There is a reasonable argument to be made that this is the most important single issue of our time.

#metascience2019

West demonstrating just how technology and it's design is not an "innocent bystander"

Search/discovery and evaluation are now inseparable

#metascience2019

West: We have, overall, a less diverse information environment over time. Ever increasing concentration in a relatively smaller proportion of the information both within and between fields.

#metascience2019

In case anyone was wondering, I just tried to buy hharmony.com. Sadly someone bought, but it's on sale for $3k if someone is really committed.

#metascience2019

Hey you at #metascience2019! Just got a questio, can you help?

"I'm really interested in getting into metascience research, and am retraining via online classes. Am I right in thinking that a thorough grounding in stats and data science is the way to start?"

Note: message edited to omit potentially identifying details.

Simine Vazire up, asking how we help make science a self-righting ship.

How do we get there? What went wrong?

"Transparency just gives us the credit that we deserve"

How do we deserve it first?

#metascience2019

Vazire: Credibility through Transparency + Quality Control + Corrections

Need ALL THREE to be a field that is credible.

#metascience2019

"It's shocking how little accountability there is for journals... I don't hear anyone talking about that."

<raises hand tentatively a little>

Shameless plug: part 1 of XvY is a systematic review of strength of causal inference in the top 10 medical journals...

#metascience2019

Goodman: "Another group that doesn't get criticized because everyone is afraid of them is funders ... Also, we need to be as critical of our own metascience as we are of everyone else"

#metascience2019

Vizine asks Who watches the watchers? We need to creating a culture which promotes critical discussion, that is internally self-correcting, otherwise we get infinite recursion.

#metascience2019

Damn, that was a FANTASTIC talk. Brutally honest, including really important personal and professional self critique.

And so ends day of #metascience2019

Apologies to Simine Vazire for multiple misspellings above. Sorry!!

Final day of #metascience2019! Starting off with a funders panel with representatives from Wellcome Trust, the Templeton World Charity Foundation, and the NSF.

Warning: it is likely that this thread is about to get very snarky in ill advised ways. Sorry, future Noah.

Arthur Lupia (NSF): New programs directly in meta science.

A good start?

However, I want to take a quick moment while each of the organizations tells us how important this all is with extremely vetted official statements for a major aside.

We are at #metascience2019 listening to major funders, whose collective failures to recognize and act on immense, existential problems in science are a major part of the reasons why we are here today.

Things are so bad that DARPA (!) has emerged as an unlikely potential savior.

Yes, that DARPA; the organization whose primary remit is as a tech accelerator for defense applications.

Five years ago, no one in this room would have touched DARPA with a 10 foot Howitzer. Today, they seem like a damn good bet. And I, for one, welcome our new robot overlords.

Lupia (NSF) is describing a very honest and fairly accurate view of the problems that young researchers in particular are facing that are horrific for science, and the researchers themselves. And that's fine. But what, specifically, is going to be done?

#metascience2019

Chonnettia Jones: If we don't do something real and serious, things are going to keep getting worse... We also don't have the evidence to act.

Me: Evidence and good science important, but waiting for "good evidence" of what to do is going to be far too little far too late.

Potgieter: "I rely on the scientists to make decisions"

This is one way that funders and journal editors alike shirk responsibility. It's not me; it's the scientists who are on the panels/citing papers/whatever; we're just following what they tell us.

#metascience2019

Questions start, EVERYONE lines up (including, of course, Player #1).

If there's gonna be fightin' words, seems likely to be in the next half hour.

#metascience2019

Back to the top-down vs. bottom up:

There are a TON of competitors in this game, all the rules and incentives are terrible, but there are only two nodes where there is a small number of people who have actual decision power: journal editors and funders.

#metascience2019

Goodman: We should be testing, but we don't know in the end. We should not be demanding more of meta science than we demand from basic science in the face of technical uncertainty, when we have every reason to believe that we can make positive change now.

#metascience2019

Look, take all this with a grain of salt as I have a major conflict of interest (just started at METRICS under Goodman and Ioannidis this week), but DAMN, every one of Goodman's comments at #metascience2019 have been ON POINT.

Really proud to be here right now.

Jones speaking up for the younger generation researchers' well-being (mental health, etc).

This is real, and important. Worth noting synergy: giving us some hope that we can actually make a living doing good science would go a long way.

#metascience2019

Now up for "Panel Discussion: Reflections on metascience topics and findings," which is consists of a smattering of science folks who have some form of fame and/or pop presence. Not sure exactly how this panel was selected, curious, except to provoke fights?

#metascience2019

Discussion thusfar has centered on psych replication. Out of my element here, but I don't love the idea that we are not in a "crisis" because we shouldn't expect (or, by implication, strive for) anything other than local effects?

Am I misunderstanding?

#metascience2019

Well, that was ... something.

Lisa Feldman Barrett: <note: editorializing> Hey young people, I was once like you thinking we should improve science, but if you should probably stop trying to tell us who are now the establishment that we're wrong.

#metascience2019

I am marking this to go back to the video later. I really hope I missed something and misunderstood, but honestly have no idea what the point of that rant was, which I will consider deeply over a round of avocado toast in house I will never own.

#metascience2019

CURTAIN PULL SLIDE TRANSITION!!!!

#metascience2019

It's worth noting that bad incentives do not necessarily imply malicious intent. All you need is variance (even fully random variance) in how rigorous work is to systematically favor research and researchers with poor methodological work over rigorous work.

#metascience2019

I am BAFFLED by so everything in this panel. There is a ton of really great arguments and discussion in there, but just so many WILD throwaway rants, and then NO Q&A???

I have nothing to say except that I have no clue what just happened.

#metascience2019

Journalists panel! Not gonna lie, I am super excited about this. All four panelists are people who I have followed for years.

Ivan Oransky: "If we could get the previous panelists back on the stage, we have some questions."

#metascience2019

These are journalists who have been successful doing REALLY good work, but through rare circumstances and/or somewhat niche outlets (maybe not niche to people in this room).

What about the science desk journalists who are in the usual mainstream grind?

#metascience2019

Aschwanden: "I am always surprised at just how much scientists are willing to hype their own work."

Confession time: I once e-mailed one of my papers to her to see if she was interested in it.

#metascience2019

Aschwanden: "Just because it's hard and expensive to do good science, that isn't an excuse for putting garbage out there ... [many of] these studies just should never have been done."

YUSSSS!

#metascience2019

Oransky: "In my experience, very little actually happens when there isn't public scrutiny," in response to the idea that we should criticize science in private but support it in the public.

#metascience2019

I agree enthusiastically with everything that is being discussed, but it's been mostly about the role of journalists and scientists/instutitions, but relatively little internal critique of the broader field of science journalism in practice.

#metascience2019

... wouldn't be a major issue for this meeting, except that poor science journalism catalyzes many of the worst systems and incentives in science.

E.g. One reason that nutritional epidemiology is so awful is because it gets press, and from press, $

#metascience2019

There we go! Heading home from the final day of #metascience2019. It was great passing notes in class with y'all. I certainly learned a ton, met amazing people, and can't wait to get working.

Lots to digest, will give some summary thoughts after some sleep.

Had a full night of sleep, time to reflect on #metascience2019.

Firstly, this thread consists of what is sticking with me, not a summary or a play by play. LOTS of threads for that, the most complete of which is from @joseph_fridman's feed.

Bottom line: 7/10.

@joseph_fridman While criticism and discomfort are going to make up the vast majority of this thread (I am, after all, I critic), it's 7/10 for a reason: the experience overall was a very positive one. I am extremely happy I was able to go and participate.

#metascience2019

@joseph_fridman Insomuch as the primary purpose of a conference is to put people with common goals and interests in the same place to discuss, collaborate, and extend networks, #metascience2019 was a success.

I met a ton of folks, had great conversations, and have new ideas and new work.

@joseph_fridman Even the critique in this thread is, itself, progress. We now have much more information to work from to understand, mitigate, and prevent our shortcomings in whatever it is that we are doing in meta-science.

#metascience2019

@joseph_fridman I loved that so many of the talks focused on science as a social process, with real live people and everything.

I also loved the consistent connection to all of the existential questions we are facing in larger society.

#metascience2019

@joseph_fridman Almost all of the talks were individually insightful and wonderful. While I can't touch on all of them (sorry), a few highlights:

Jacob Foster's wild ride marrying social theory on knowledge production with illustrative models was a DELIGHT.

#metascience2019

@joseph_fridman Michele Nuijten not only highlighted just how astoundingly common basic statistical errors are, but also demonstrated something that was relatively rare: a concrete actual real world publicly available implemented solution.

#metascience2019

@joseph_fridman Ed Miguel's perspective on how economics, a field that is in many respects very isolated, has had continued success in reforming its science and pulling from outside ideas, in part due to a culture of rigor and reform.

#metascience2019

@joseph_fridman Dorothy Bishop and Fiona Fidlers's dives into how scientists own misconceptions and misunderstandings about statistics and science itself are rooted in social and psychological processes were both cutting and lovely to absorb.

#metascience2019

@joseph_fridman Simine Vazire's discussion about how we can can make critique and correction and bolster our self-correction mechanisms had a level of personal and professional humility that was deeply captivating.

#metascience2019

@joseph_fridman Tim Errington pulled back the curtain what it's practically like to actually try to do replication and follow-up post-publication, and some of the unexpected and damning problems encountered along the way.

#metascience2019

@joseph_fridman Carl Bergstrom showing how the fundamental structures of our scientific rewards system generates a result that is literally worse than random lotteries was fun, compelling, and deeply depressing.

#metascience2019

@joseph_fridman And the journalists panel, from four folks whose work has and continued pushed science perhaps even more than almost any scientist in that room was honest, brutal, and wonderful.

And so, so many more, and I am truly sorry I can't list all of them.

#metascience2019

@joseph_fridman But while the talks were INDIVIDUALLY fantastic(with some notable exceptions, which I will discuss), collectively they were far, far less fantastic.

The collective issues with #metascience2019 represent and potentially perpetuate some of the biggest problems in science itself.

@joseph_fridman Let's not beat around the bush here: This was a conference organized by five well-established semi-famous white guys, and the conference was a reflection of their priorities and social structures.

#metascience2019

@joseph_fridman If there is anything to understand about meta-science, it is how individually-justifiable decisions contribute to and reinforce systematic problems.

The increasingly concentrated boy's club nature of science is a major problem both in science and in #metascience2019

@joseph_fridman Several of the talks highlighted how people were differentially sorted along the research pipeline along gender lines, and how that impacts scientific production. Gender lines that look (WEIRD!) like the ones representing/organizing the conference.

#metascience2019

@joseph_fridman While there was some (really great) perspectives on and from gender, there was EXTREMELY little with regard to race.

But the one thing that I noticed the most (perhaps because I personally feel it) was along seniority and generational divides.

#metascience2019

@joseph_fridman Now, there were quite a few folks who did a DAMN good job of understanding the system which scientists of our generation exist.

But wouldn't it be better if, maybe, like, ONE junior scientist got to "reflect" on things?

#metascience2019

@joseph_fridman There was even an excellent place to put it!: "Reflections on metascience topics and findings"

BTW, I still have no idea what to make from that panel, except to put on display the egotistical delusional bullshit that we're going to face to get anything done.

#metascience2019

@joseph_fridman In terms of topics covered, it would be fair to have called this whole thing a replication in psych symposium.

The phrase metascience became popular when the replication "crisis" became popular, with huge amounts of funding being thrown in that direction.

#metascience2019

@joseph_fridman On the one hand, it's critical to get more people and resources turned towards problems in science, and psych in replication is a vehicle toward getting that moving.

However, I worry that we're reinforcing many of the problems that got us here to start with.

#metascience2019

@joseph_fridman The lack of systematic and usable understanding of the strength and utility of science + reliance on null hypothesis testing is a major part of what got us to the replication "crisis."

And replication (in its current form) is just more null hypothesis testing.

#metascience2019

@joseph_fridman By concentrating on replication, to the near exclusion of everything else, we may be reinforcing bigger, more fundamental problems.

And there are just SO many other problems we could have been dealing with instead of replicating talking about replication.

#metascience2019

@joseph_fridman Maybe starting with replication and moving forward from there as we get more momentum is the right strategic choice? But I honestly doubt that was a conscious decision, and get the feeling that the organizers chose from within their wheelhouse.

#metascience2019

@joseph_fridman #metascience2019 did a good job picking topics that highlighted the depth, severity, and degree of entrenchment of institutional problems.

Not great for the sanity of anyone starting out a career in science, but good to have reality in front of you if you want to change it.

@joseph_fridman Personally, I've more or less given up hope that our generation is going to actually be able to do even a minimally reasonable level of rigorous research. I'm sticking out the fight to change it, in the hopes that the next generation gets to.

#metascience2019

@joseph_fridman But it's easier for me to say that, given that I have a lot of structural advantages (few personal responsibilities, lots of outside options should I fail, etc.). I can't imagine what it's like for others who aren't so lucky.

#metascience2019

@joseph_fridman Back to topics that weren't represented: Where were all the journal editors? Why didn't we have a panel involving representation of JAMA and Science and the gang? If there is anyone who has actual power to change things, it's those folks.

#metascience2019

@joseph_fridman The funder's panel was ... interesting. Where was NIH???

Had a lot of feelings about the NSF rep. Have to have a grain of salt, any representation they send to a meta science conference is going to be friendlier than average to such a conference's ideas.

#metascience2019

@joseph_fridman Firstly, he really did sound like he understood the problems (with one notable and infuriating exception). But I do felt like we got the placating response (we made a brand new <teensy> fund for this kind of research, yay!). I guess that's something, at least.

#metascience2019

@joseph_fridman It's tough to feel good about the future when it looks like we're hitting the wrong problems, without much if any buy-in from the people and places that could actually have impact on the right problems.

#metascience2019

@joseph_fridman To close out: on the one hand I feel ready, excited, and motivated to get working and fighting alongside a lot of amazing folks.

On the other, I just don't know if we've got what it takes if this is the best we can do. Hope we can do better.

#metascience2019