Thread by @lymanstoneky

Apropos @elonmusk 's demography comments and an exchange with @ikashnitsky , it's interesting to ask, "Just how reliable are major global demographic databases?"

My view: not very!

This is not because the demographers at UNPD or IHME are doofuses or something; it's because producing big global forecasts is extremely difficult even when the data is all homogenous and reliable. But sorting through competing data sources that disagree is even harder!

Infamously, very smart demographers disagree about China's population to the tune of tens or hundreds of millions of people, and China's TFR is debated to be anywhere from 0.9 to 1.7 in recent years! Right there, that tells you global forecasts are gonna be dicey.

But while the data for China is infamous, we can give folks at IHME or UNPD or elsewhere a pass on that because *the data is extremely contested*.

The more interesting example to consider is India.

Here are estimates of births in India from the WPP2019 from UNPD, IHME's 2017 model, and India's Civil Registration System (CRS). The CRS data has two lines: one for the raw number of birth registrations with occurence in year X, a second for the CRS' best guess at "true totals."

India's birth registration reports are very readable and accessible. Here's the 2019 report: censusindia.gov.in/2011-Common/CR…

We know for a fact that India's birth registration system is *underreporting* births. Why? Because some districts still don't even have the system set up at all, and in some districts birth registration implies birth rates which are extremely implausibly low.

Beyond that, India conducts regular surveys where they ask mothers if their child's births were registered. Large shares say no. The share of kids under age 5 whose birth was registered in these surveys rose from 41% to 2005, to 80% in 2015, to 89% in 2020.

So, keep this crucial fact in mind:

We *know* that in India, the "official number of births" is SMALLER THAN the true number of births.

Both because surveys show large rates of non-registration, and because we see clear spatial gaps and implausibly low registration totals in many areas of the country.

Next keep in mind another important fact:

We do *not* know exactly how reliable India's past censuses are, nor any of its other demographic data. Probably the census data is decently good! But it's not perfect.

Okay, so, what do UNPD and IHME say about births in India?

Well, here's a graph of UNPD and IHME births in India, as well as India's crude number of birth registrations, and the CRS estimate of "adjusted" registrations.

The first key thing to notice is that IHME, UNPD, and CRS all disagree about *historic* births. They disagree in level by several million births, but even in trend, with some sources showing different bumps and dips.

In recent years their differences become *especially* pronounced.

And in 2019, an odd thing happens.

The big global demographic forecast programs estimated fewer births *than were actually registered*.

See that last 2019 line? That's showing that just crude registrations, WHICH WE KNOW ARE INCOMPLETE, were higher than the estimates of UNPD and IHME. True births must be even higher.

Okay, so they messed up births.

But sometimes you mess up one thing and get the rest right.

But not so here.

Here's the UN and IHME estimates of TFR vs. various other direct data sources. You'll note there is considerable disagreement at times between sources! Broadly, "direct" fertility questions often seem to underestimate vs other methods.

There are decent reasons to think the Sample Registration System is the best estimate of fertility. The way this system works is that the government hires "enumerators" in about 9,000 neighborhoods or villages around India. Those enumerators are paid to record every birth/death

In their district. In total, about 8 million Indians live within the enumeration districts, so probably about 3 million reproductive-age women "in-sample." They also conduct a mid-year evaluation survey to ensure correctness. It's a very neat system more poor countries...

.... should consider emulating as a way to get good vital data without necessarily having to build an entire vital registration system in one go.

On the other hand, we have the DHS/NFHS surveys. These are very big surveys. The NFS-5 included about 750,000 women who filled out detailed, in-depth surveys with interviewers.

I won't litigate why we should prefer one source or another, I'll just note that we have two evidently very high-quality sources which disagree.

In recent years, UNPD takes the SRS... but not in past years, where they take DHS/Census reconstructions.

Again, just to highlight the complexity here, UNPD jumps between three different sources of TFR *at least*. That may be prudent, but it's important to understand how many researcher degrees of freedom go into producing population estimates.

Okay but anyways, in recent years, let's assume the SRS data is correct.

If the SRS data is correct about births-per-woman, and UNPD more-or-less adopts it, but UNPD birth estimates are *too low*, that means the UN estimates of *women* must *also* be too low.

Since "Total Births = Births per Woman X Woman", if you've undershot total births and correctly estimated Births per Woman, you must have also undershot "Number of women."

Which means you've gotta raise an eyebrow at the core population numbers!

If instead of SRS the DHS fertility numbers are correct it's an even dicier situation: UNPD TFR is considerably above the DHS estimates, so you'd be overestimating Births per Woman, underestimating Total Births, which means you'd be HUGELY underestimating "Number of Women"

Which is to say: while @elonmusk 's disdain for UNPD is probably a bit overblown, at the end of the day even among smart demographers looking at countries with multiple high-quality data systems, it's not clear what the real data should be!