Profile picture
Nate Silver @NateSilver538
, 3 tweets, 1 min read Read on Twitter
1. Going through some old data/code I last worked on 8 years ago. One thing I've learned since then is that when combining different datasets or doing other complicated data processing, it pays to be really anal-retentive about missing data or data that doesn't pass sanity checks
2. Say you have a sample size of 5,000 cases, and of those cases, a dozen or so are missing or look as though they're miscoded. You might think, "That's not bad. I'll just throw those cases out and I'll be fine". And you probably will be fine if that's all there is to it.
3. But more often than you might think, the missing/miscoded/"outlier" cases indicate a larger, more systematic problem with your code or with your data. My advice is to do due diligence on those cases BEFORE moving on to next steps, because errors tend to compound.
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Nate Silver
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($3.00/month or $30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!