In "Psychology’s Increased Rigor Is Good News. But Is It Only Good News?" Barry Schwartz concludes "My aim here has only been to encourage us to acknowledge that there is a price."

I agree with that, but disagree with most of the rest of the analysis. behavioralscientist.org/psychologys-in… Area of agreement: We must examine the impact of new behaviors to improve rigor because there are almost always unintended consequences that might be counter to the aims of accelerating progress.

Everyone moving from twitter to mastodon is very likely to fail because it is a collective action problem, benefits depend on others actions.

Here's how it can succeed. It will require your commitment for 1 month to do a few things. Read and retweet if you commit to do them.

https://twitter.com/BrianNosek/status/1587024198567952386

Context: You might be tempted to post new content on both platforms until it is clear that Mastodon is going mainstream. This WILL NOT work.

Most user behavior is consuming content, not producing it. Twitter has built-in advantage of inertia and audience.

I favor an all Green OA world with peer review, typesetting, copyediting, etc as microservices, but I don't see an all Gold OA world as being necessarily as bad as others do. A few reasons and would love to have these challenged by others who are thinking about system change... In an all Gold OA world, price is evident and meaningful for decisions of authors for where to submit. As such, academic publishing becomes an actual marketplace in which the decision-making consumer is price conscious. Therefore, competition on price increases.

Baumeister, Tice, & Bushman have a new piece examining what we have learned from multi-site replication studies here: researchgate.net/profile/Brad-B…

It is worth taking seriously even though I don't agree with all the conclusions.

A thread & relevant paper: annualreviews.org/doi/abs/10.114… The positives: The piece has no invective, no misattribution of claims, and represents other perspectives fairly.

You might counter that is a low bar. For hot topics, I disagree. Also, compare the piece with responses to replication circa 2014-2016. This is real, scholarly work.

Massive status bias in peer review.

534 reviewers randomized to review the same paper revealing the low status, high status, or neither author. 65% reject low status, 23% reject high status.

Amazing work by Juergen Huber and colleagues. #prc9 Or, look at it another way. If the reviewers knew only the low status author, just 2% said to accept without revisions. If the reviewers knew only the high status author, almost 21% said to accept without revisions.

Lovely replies on the upsides.

In case it is useful perspective for anyone else, here's part of how I managed the downsides as an ECR so that the upsides dominated my experience in academia.

https://twitter.com/westwoodsam1/status/1522491798949609472

Key downsides that needed managing for me: (a) dysfunctional culture that rewarded flashy findings over rigor and my core values, (b) extremely competitive job market, and (c) mysterious and seemingly life-defining "tenure"

Across my research on implicit bias, morality, ideology, reproducibility, and open science, I have been on the receiving end of a lot of scholarly criticism. It has been a gift to my career. But, not all criticism is equal.

My primary insight as a recipient... Besides decency, there are strong self-interested reasons for critics to treat the subject of their criticism as well as possible.

Three points from what might be the most obvious to the somewhat less obvious:

How does one stay motivated about academic work with the long delays and external influences of getting rewards? Here are some of my favorite strategies.

Short thread.

(Elaborating on a comment in a separate thread.) It is great to celebrate wins (grants, publications), but they are partly outside of my control. I increase predictability and controllability of rewards by celebrating milestones that I control -- completion of data collection, posting a preprint, submitting the grant.

Parents: Taste-testing is a great way to engage kids on science and how methodology can address potential biases.

Here's an example w/@LastCrumbCookie we did recently. This is after a few years of lots of simpler taste tests. Most of our prior taste tests were amenable to some blinding such as comparing fast food chicken and fries and testing generic vs. brand-name.

Haven wanted to do @LastCrumbCookie and compare their 12 varieties. Blinding not possible & 12 is a lot to test! Challenging for design.

For the last 2.5 years, my daughters and I have been rating breakfast places in #charlottesville #cville area. We rated 51 restaurants from 1 (worst) to 10 (best) on taste, presentation, menu, ambiance, & service. We also recorded cost-per-person.

Here's what we learned. 1/ Across 51 restaurants we spent $1,625.36 pre-tip, average cost of $9.91/person (sometimes other family members joined).

Cheapest per person: Duck Donuts $3.10, Sugar Shack $3.41, Bojangles $4.30.

Most expensive per person: The Ridley $27.08, Farm Bell Kitchen $17.81, Fig $17.44

Sharpen your intuitions about plausibility of observed effect sizes.

r > .60?

Is that effect plausibly as large as the relationship between gender and height (.67) or nearness to the equator and temperature (.60)? r > .50?

Is that effect plausibly as large as the relationship between gender and arm strength (.55) or increasing age and declining speed of information processing in adults (.52)?

An ironic back story (h/t @Edit0r_At_Large for evoking the memory).

This paper was itself submitted as a Registered Report in 2018 but was rejected!

Reviews were excellent. Identified some limitations we could solve, others we would have needed to debate on feasibility grounds.

https://twitter.com/BrianNosek/status/1408081726044319749

@Edit0r_At_Large The journal did invite a resubmission if we wanted to try to address them. However, we ultimately decided not to resubmit because of timing. We had a grant deadline to consider.

We did incorporate reviewer suggestions that we could into the final design and proceeded.

New in Nature Human Behavior: We had 353 peer reviewers evaluate published Registered Reports versus comparison articles on 19 outcome criteria. We found that RRs were consistently rated higher on rigor and quality.

Paper nature.com/articles/s4156…

Green OA osf.io/preprints/meta… Figure shows performance of RRs versus comparison articles on 19 criteria and 95% credible intervals. Red criteria evaluated before knowing the results, blue after knowing the results, green summarizing whole paper.

10 years of replication and reform in psychology. What has been done and learned?

Our latest paper prepared for the Annual Review summarizes the advances in conducting and understanding replication and the reform movement that has spawned around it.

psyarxiv.com/ksfvq/

1/ Co-authors: @Tom_Hardwicke @hmoshontz @AllardThuriot @katiecorker Anna Dreber @fidlerfm @JoeHilgard @melissaekline @MicheleNuijten @dingding_peng Felipe Romero @annemscheel @ldscherer @nicebread303 @siminevazire 2/

Our prospective replication study released!

5 years: 16 novel discoveries get round-robin replication.

Preregistration, large samples, transparency of materials.

Replication effect sizes 97% the size of confirmatory tests!

psyarxiv.com/n2a9x

Lead: @JProtzko 1/ When teams made a new discovery, they submitted it to a prereg’d confirmatory test (orange).

Confirmatory tests subjected to 4 replications (Ns ~ 1500 each)

Original team wrote full methods section. Team conducted independent replications (green) and a self-replication (blue).

My Jewish spouse loves Christmas. Okay, fine, who doesn’t?

Converting to a “Valentine’s tree” seemed a bit excessive.

Came home from a trip tonight. Dear god, how do I make it stop? Came downstairs and the tree is transformed with whoopee cushions, fake(?) poop, and cockroaches all-over it.

I am stuck in the house with an April Fools Tree and the family members responsible for it.

Happy to elaborate. Think of preregistration of analysis plans as hypothesizing, data analysis, and scenario planning all rolled into one and without knowing what the data are. This creates a novel decision-making situation. 1/

https://twitter.com/Maxyusc/status/1132226934690029568

For example, the first time preregistering an analysis plan, many people report being shocked at how hard it is without seeing the data. It produces a recognition that our analysis decision-making (and hypothesizing) had been much more data contingent than we realized. 2/

Some predictions about whether the researcher's ideology effects their likelihood of replicating a prior result. ht @jayvanbavel

First, I have no doubt that ideology CAN influence replicability. Classic Rosenthal work + more provides good basis.

So, under what conditions? 1. Ideology may guide selection of studies to replicate. More likely to pursue implausible X because it disagrees with my priors; and pursue plausible Y because it agrees with my priors.

On balance, this may be a benefit of ideology to help with self-correction and bolstering.

Many Labs 2: 28 findings, 60+ samples, ~7000 participants each study, 186 authors, 36 nations.

Successfully replicated 14 of 28 psyarxiv.com/9654g

ML2 may be more important than Reproducibility Project: Psychology. Here’s why...

@michevianello @fredhasselman @raklein3 ML2 minimized boring reasons for failure. First, using original materials & Registered Reports cos.io/rr all 28 replications met expert reviewed quality control standards. Failure to replicate not easily dismissed as replication incompetence. psyarxiv.com/9654g

We replicated 21 social science experiments in Science or Nature. We succeeded with 13. Replication effect sizes were half of originals. All materials, data, code, & reports: osf.io/pfdyw/, preprint socarxiv.org/4hmb6/, Nature Human Behavior nature.com/articles/s4156… Using prediction markets we found that researchers were very accurate in predicting which studies would replicate and which would not. (blue=successful replications; yellow=failed replications; x-axis=market closing price) socarxiv.org/4hmb6/ nature.com/articles/s4156… #SSRP