Profile picture
Julia Rohrer @dingding_peng
, 7 tweets, 2 min read Read on Twitter
So over the course of the last years, I've developed a certain knee-jerk reaction where I'm super cautious about p-values between (~).01 and .05.>
I've been working with a few large-scale data sets and one thing I've experienced is that effects that are robust and are likely to pop up in other data sets as well come with much lower p-values.>
Example from our specification curve paper (home.uni-leipzig.de/diffdiag/pppd/…): I get super tiny p-values, almost regardless of how I slice and dice my data.>
Now in contrast, if there's nothing going on, there is nothing going on. But notice that the "just significant" values just under .05 occur just as often as they do when there is strong evidence for an effect.>
So if I run some analyses and end up with p = .03, I'll initially assume that I've encountered the 2nd situation (nothing going on) and double check whether the "finding" holds up or replicates in another sample.>
That's obviously just an intuition, there are formal arguments to be made here as well, and things look quite different if the analytic plan has been exactly pre-specified! But I still consider it one of the more helpful gut feelings.>
And needless to say, smart people such as @siminevazire have already blogged about this: sometimesimwrong.typepad.com/wrong/2015/06/…
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Julia Rohrer
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($3.00/month or $30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!