Thread by @SBF_FTX

1) Don't let the drive for a narrative overcome your priors; aka the slow, sad decay of fivethirtyeight.

fivethirtyeight.com/features/justi…

2) This article is dumb.

It is centered around the observation that, this season, Lamar Jackson has been better on early downs while Justin Herbert has been better on late downs.

Why is this????

3) Sample sizes, mostly.

The sample sizes so far are pretty small -- around 60 3rd down attempts per team. The difference between the Ravens and Chargers completion % is about 14% (35% vs 49%); the standard deviation due to raw luck here is ~40%/sqrt(60)*sqrt(2) ~ 7%

4) So this is a 2 standard deviation effect. Big, right?

Not really. There are 15 games this week; just choosing the largest discrepancy from those games would yield a result this large on average.

And looking specifically at this game, there are a ton of possible factors.

5) But more damningly, there's no attempt to justify this as a likely factor. What makes the priors on it high enough to overcome p-hacking?

6) The thing is, writing constant sports columns is *hard*.

In the end, what's the "right" article on the Chargers vs Ravens?

Probably something like "idk, they're both good, Lamar Jackson might lead the game in rushing, though I guess Austin Ekeler could too."

7) And that article is boring.

But to write a better-than-boring article, you have to say more than the average sports fan would know about a game, and those new facts have to be interesting and meaningful.

That's fucking hard.

8) That's what teams hire departments to try to do--come up with actionable analyses that are better than public ones--and combine with proprietary data and tracking.

It's not impossible--it's just hard. To write great content, you have to be great at it, and try really hard.

9) Relative to the status quo, that accurately describes fivethirtyeight circa 2008: the best public resource for election modeling.

And, maybe, 2012.

But not 2021.

10) Over time, like so many things, its content has become a random milieu of smart-but-not-smart-enough takes, and models that aren't quite good enough to beat common sense.

See, e.g., their 2020 election model--not terrible, but worse than "meh, < 50%, but at least 25%".

11) It's been 8 years since fivethirtyeight's presidential election model was more accurate than the in-house models I helped build, and the rate of real groaners has skyrocketed (fivethirtyeight.com/features/how-i…).

The sports journalism, meanwhile, has been even worse.

12) Anyway, I guess this is another thing to file the ever growing list of "things I hope never happen to FTX".

13) In the end, I now see myself has having two core roles:

a) Make sure that in the end we make the right decisions and do the right things; "the buck stops here"

b) don't let institutional inertia decay awa our ability to operate effectively