Thread by @tylerblack32

1/ VERY SERIOUS, VERY COMPLEX DISCUSSION AHEAD:

The @FAIRHealth White Paper showing MH insurance claims in kids is being reported deceptively.

WRONGLY (!!!), people are using it as evidence that "Mental health claims are up in kids."

Why is that wrong? Let's break it down.

2/ What is it?

It's an insurance report that does a breakdown of 32 BILLION claim records, looking at month-to-month changes Jan-Nov 2019 vs 2020. It focuses on pediatric (13-18) mental health insurance claims:

* overall
* self harm
* overdoses
* diagnoses

3/ This very important part of the methodology is at the heart of the confusion on how this paper is being reported.

It is NOT looking at raw #'s increasing, it's looking at percentages of overall claims.

In other words, what % of claims were for mental health.

4/ Why do I say it's confusing and deceptive?

When you look at this graph, it LOOKS like mental health claims for kids went up, while ALL claims went down.

5/ But in reality, the MENTAL HEALTH (green) portion of the graph is the year over year change AS A PERCENT OF THE TOTAL CLAIMS, NOT the year over year change. The purple graph, overall, is year-over-year change.

THE GREEN AND PURPLE BARS ARE NOT ON THE SAME AXIS.

6/ The classic orange and apple thing I've been discussing:

If I start with 15 apples and 5 oranges, I have 20 pieces of fruit together.

25% of my fruit is oranges.

7/ If I lose 11 apples, and 1 orange (BOTH DECREASED), I now have 4 apples and 4 oranges. 8 fruit.

8/ So here we have the @FAIRHealth situation. Both apples and oranges decreased. (20 --> 8 = 60% reduction in fruit). However, with 4 apples and 4 oranges, 50% of the fruit are now oranges.

So oranges went from 25% to 50%. That's a 100% increase in the orange % of fruit!!

9/ Look at the pivot in the language, without explanation! In the first paragraph, it's an increase "as a percentage of all medical claim lines".

In the 2nd paragraph, it's quickly morphed into "increased mental health claim lines".

This is CONFUSING, and ACTUALLY DECEPTIVE.

10/ And how is it being reported? Well, the lazy use of language held. Check out all the media attention it's getting.

First, @axios did not detect the deceptive statistics. Author @caitlinnowens repurposed the graph in axios colors, repeating the error in different axes.

11/ Same with @foxnews.

Incorrectly, it that "mental health claims doubled" when it's the "a doubling of the percentage of all claims (which were severely down) that were mental health claims," a less sexy mouthful for sure.

12/ @paulnestadt is quoted (I respect you a lot and love your work!!), and either wasn't given the time to, or didn't detect this statistical deception - it is not an increase in claims (nor CDC data an increase in ER MH presentations), its the increase in proportion.

13/ The venerable @TIME got it wrong too. Multiple times, they confuse the language. "claims for any mental health disorder increased by about 100% for the 13-18 group."

14/ Throughout the document, its the same statistical error. MH Claims are reflected as a "% CHANGE FROM 2019-2020 AS A PERCENTAGE OF OVERALL CLAIMS" in the green, and overall claims as a "% CHANGE FROM 2019-2020" in the purple.

15/ Never fear, we can do math!

In April 2019, intentional self harm claims were 0.0137% of all claims. In April 2020, they were 0.0274% of all claims.

This is the "increase of 99.83%" that is being touted in the media. From 0.0137% to 0.0274%, up 99.83%!

16/ BUT WAIT! In April 2020, we also know that ALL CLAIMS were OVERALL DOWN by 53.4%!!!

17/ So imagine there were 1,000,000 total claims in April 2019. 0.0137% of those were for self-harm, so that's 137 self-harm claims.

In April 2020, claims were down by 53.4%! That means there were 466,000 in April 2020. 0.0274% of those were for self harm. That's 128 claims.

18/ PRECISELY the apples-oranges scenario I laid out before.

If we start in April 2019 with 1,000,000 insurance claims:

In April 2019, it was 137 cases of self harm.
In April 2020, it was 128 cases of self harm.

A REDUCTION OF 6.5% (but 99.5% increase in "percentage")

19/ This is precisely the type of statistics that require expert analysis and I am absolutely shocked that @FAIRHealth has not clarified this.

20/ I suspect, year over year, mental health claims DECREASED, but it is only the % compared to the other claims that has increased. This would be consistent with national data, like the often-misquoted @CDCgov data.

21/ Most of the experts quoted in the news pieces on this study did not detect this, because it requires a CAREFUL reading and analysis of the language and graph, but also because the GRAPHS WERE DECEPTIVE.

22/ IT IS NEVER OK to two different things on one axis.

"% change year over year" IS NOT THE SAME as "% of change year over year as a percentage of all cases".

23/ Whoever made this report for @fairhealth may not have recognized this error, but they CERTAINLY must have recognized how it was being misreported.

24/ Now @fairhealth could EASILY clear this up by simply answering the following question:

What were the #'s of RAW CLAIMS for mental health in 2019 compared to 2020 for the same categories?

25/ In their preamble, they state they COULDN'T report raw MH claim numbers because the # of people insured changed. But then, they DO EXACTLY THAT!

They compared "year over year changes in ALL claims," despite the denominator changing, in almost EVERY GRAPH!

26/ If the denominator changes, you have to simply say that.

"We can't compare them fully because it is comparing different, non-random samples of the population."

Or, provide context: "Claims were down in mental health, but they were down less than non-mental health claims."

27/ In my jurisdiction (BC), in CDC data across the US, the same trend holds. OVERALL, cases in mental health are down or about the same for kids, they are just DOWN LESS than medical cases, so the "proportion" has increased.

28/ This is probably the longest tweet-thread I've ever done, but it's also infuriating to me that such national attention was given to such a statistically flawed presentation.

29/ I hope @fairhealth cares FAIRLY about HEALTH data, and corrects the error in axes in all of its graphs, and corrects the record to how its report is being reported.

30/ P.S. Americans have still tied health care to employment and private insurance. That should probably change too.