Sharpen your intuitions about plausibility of observed effect sizes.

r > .60?

Is that effect plausibly as large as the relationship between gender and height (.67) or nearness to the equator and temperature (.60)?
r > .50?

Is that effect plausibly as large as the relationship between gender and arm strength (.55) or increasing age and declining speed of information processing in adults (.52)?
r > .40?

Is that effect plausibly as large as the relationship between weight and height (.44), gender and self-reported nuturance (.42), or loss in habitat size and population decline (.40)?
r > .30?

Is that effect plausibly as large as the relationship between elevation and daily temperature (.34), viagra and sexual functioning (.38), past behavior predicting future behavior (.39), or sleeping pills and insomnia reduction (.30)?
r > .20?

Is that effect plausibly as large as the relationship between marital relationship quality and parent-child relationship quality (.22), alcohol and aggressive behavior (.23), or gender and weight (.26)?
r > .10?

Is that effect plausibly as large as the relationship between antihistamine and runny nose (.11), childhood lead exposure and IQ (.12), anti-inflammatories and pain reduction (.14), self-disclosure and likability (.14), or nicotine patch and smoking abstinence (.18)?
r > .00?

Is that effect plausibly as large as the relationship between aspirin use and death by heart attack (.02), calcium and bone mass premenopausal (.08), gender and observed risk taking (.09), or parental divorce and child well-being problems (.09)?
Source: Meyer et al (2001) [doi.apa.org/doiLanding?doi…]

h/t to @williamspsych and @suzannecarrie for recalling and finding it.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Brian Nosek

Brian Nosek Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @BrianNosek

25 Jun
An ironic back story (h/t @Edit0r_At_Large for evoking the memory).

This paper was itself submitted as a Registered Report in 2018 but was rejected!

Reviews were excellent. Identified some limitations we could solve, others we would have needed to debate on feasibility grounds.
@Edit0r_At_Large The journal did invite a resubmission if we wanted to try to address them. However, we ultimately decided not to resubmit because of timing. We had a grant deadline to consider.

We did incorporate reviewer suggestions that we could into the final design and proceeded.
We eventually had the full report and that was peer reviewed in the normal process.

We published the paper in Nature Human Behaviour.

The RR was originally submitted to Nature Human Behaviour.

I think the RR submission did meaningfully improve our design & odds of success.
Read 9 tweets
24 Jun
New in Nature Human Behavior: We had 353 peer reviewers evaluate published Registered Reports versus comparison articles on 19 outcome criteria. We found that RRs were consistently rated higher on rigor and quality.

Paper nature.com/articles/s4156…

Green OA osf.io/preprints/meta…
Figure shows performance of RRs versus comparison articles on 19 criteria and 95% credible intervals. Red criteria evaluated before knowing the results, blue after knowing the results, green summarizing whole paper. Image
Congrats to @cksoderberg Tim Errington @SchiavoneSays @julia_gb @FSingletonThorn @siminevazire and Kevin Esterling for the excellent work on this project to provide an additional evidence base for how Registered Reports can alter the credibility of published research.
Read 6 tweets
9 Feb
10 years of replication and reform in psychology. What has been done and learned?

Our latest paper prepared for the Annual Review summarizes the advances in conducting and understanding replication and the reform movement that has spawned around it.

psyarxiv.com/ksfvq/

1/
We open w/ anecdote of the 2014 special issue of Social Psychology. The event encapsulated themes that played out over the decade. The issue brought attention to replications, Registered Reports, & spawned “repligate”

econtent.hogrefe.com/toc/zsp/45/3

Figure from royalsocietypublishing.org/doi/full/10.10…
Read 15 tweets
10 Sep 20
Our prospective replication study released!

5 years: 16 novel discoveries get round-robin replication.

Preregistration, large samples, transparency of materials.

Replication effect sizes 97% the size of confirmatory tests!

psyarxiv.com/n2a9x

Lead: @JProtzko 1/
When teams made a new discovery, they submitted it to a prereg’d confirmatory test (orange).

Confirmatory tests subjected to 4 replications (Ns ~ 1500 each)

Original team wrote full methods section. Team conducted independent replications (green) and a self-replication (blue).
Based on confirmatory effect sizes and replication sample sizes, we’d expect 80% successful replications (p<.05). We observed 86%.

Exceeding possible replication rate based on power surely due to chance. But, outcome clearly indicates that high replicability is achievable
Read 10 tweets
18 Feb 20
My Jewish spouse loves Christmas. Okay, fine, who doesn’t?

Converting to a “Valentine’s tree” seemed a bit excessive.

Came home from a trip tonight. Dear god, how do I make it stop? Image
Came downstairs and the tree is transformed with whoopee cushions, fake(?) poop, and cockroaches all-over it.

I am stuck in the house with an April Fools Tree and the family members responsible for it. Image
When my spouse commits, she commits.

Me? I’m starting to have my doubts. Image
Read 26 tweets
25 May 19
Happy to elaborate. Think of preregistration of analysis plans as hypothesizing, data analysis, and scenario planning all rolled into one and without knowing what the data are. This creates a novel decision-making situation. 1/
For example, the first time preregistering an analysis plan, many people report being shocked at how hard it is without seeing the data. It produces a recognition that our analysis decision-making (and hypothesizing) had been much more data contingent than we realized. 2/
Without the data, there is a lot of new mental work to articulate precisely what the hypothesis is and how the data could be used to evaluate that hypothesis. My odd experience was believing that I had been doing that all along, w/out realizing that I used so much discretion. 3/
Read 12 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(