Thread by @lorenanthes

1/ 🧵 Looks like it may be a good time to start a *live* thread on the state's recently released Evaluation Design of the work requirement waiver. 🧵

Here's a copy of the design executive summary: medicaid.ohio.gov/Portals/0/Reso…

2/ Here's what we wrote when the waiver was approved:
communitysolutions.com/medicaid-work-…

There are three things that the state is required to develop:
1) Implementation Plan
2) Monitoring Protocol
3) Evaluation Plan

This evaluation plan is OPEN FOR PUBLIC COMMENT until August 26th.

3/ I would recommend that any health journos out there that this is a fundamentally important document for covering the implementation of the work requirement waiver.

This will determine what data is used and what is made available, publicly, and at what interval.

4/ What we said we wanted in an evaluation plan:
- Issues in administration
- Coverage levels
- Health outcomes
- Financial information (impact on local govts)
- Regular releases of data available on at least county level

...just to name a few.

5/ Methodology of the state:

Quantitative and qualitative, using survey, focus group, interview and admin data through 2025.

6/ The comparison will be between expansion adults who won't be exempt and parents who have no other exemption other than being a parent.

The populations differ in key ways, particularly work/marital status, but the state will adjust as things unfold.

7/ Timeline in graphic.

OF NOTE: @GRCOhio will be the state's independent evaluator.

They have a history with ODM, but this is a good approach and an independent evaluator is something we called for in comments.

@GRCOhio 8/ Now the meat...

The data plan will include info on participants, even if they lose coverage. The qualitative data will be based on sex, employment, demographics and geographics resulting in 32 distinct research groups.

@GRCOhio 9/ Law requires that there be hypotheses studied. The research questions are tied to each of these hypotheses that the waiver will

1) Increase employment
2) Raise avg income
3) Increase commercial insurance coverage
4) Improve outcomes

@GRCOhio 10/ "Increase employment" questions:

1) Are waiver enrollees more or less likely to be employed?
2) Do enrollees who first participate in other qualifying activities more likely to be employed?
3) Types of jobs?
4) Impact on educational attainment?

@GRCOhio 11/ "Raise income" questions:

1) Are incomes higher?
2) Do other expenses change (housing, transportation, food)?
3) Are changes sustained overtime?
4) Coverage loss?
5) Medicaid debt?

@GRCOhio 12/ "Commercial insurance" questions:

1) More private coverage (ESI and Marketplace)?
2) Employment tied to ESI?
3) Take up rates of ESI when offered?
4) ESI sustained?
5) Afford ESI?
6) More likely to get Marketplace coverage?

@GRCOhio 13/ "Commercial insurance" questions con't:

7) Type of plan comprehensive (ie - do they buy junk insurance)?
8) Do family members lose coverage?
9) If they lose coverage, what are the barriers?

@GRCOhio 14/ "Outcomes Improve" questions:

1) Are outcomes better?
2) What are the trajectories of outcomes?
3) If people don't comply, what's the relationship to outcomes?
4) What types of activities do people engage in to maintain coverage?
5) How do activities change over time?

@GRCOhio 15/ "Outcomes Improve" questions con't:

6) What are common barriers to compliance?
7) Did beneficiaries get the support they needed to comply (job assistance, child care, etc)?
8) Do beneficiaries understand the requirement?
9) How many have to actively report?

@GRCOhio 16/ "Outcomes Improve" questions con't:

10) What is the state doing to reduce beneficiary burden?
11) Rate of "good cause" exemptions?
12) Why beneficiaries disenroll
13) For those who disenrolled bc of non-compliance, are the more likely to re-enroll?

@GRCOhio 17/ That's the extent of the summary.

Note there are a number of questions that require approval from other state agencies.

@GRCOhio 18/ In addition to the summary, there is a longer form proposal:
medicaid.ohio.gov/Portals/0/Reso…

@GRCOhio 19/ They are going to use a Chi-Square test for reliability. What is that, you ask? statisticssolutions.com/chi-square-tes…

Basically, are the hypotheses of the state true when we look at those who are enrolled in the waiver v. those who are similar but not enrolled.

@GRCOhio 20/ They won't use the Chi for all groups. Some, they'll use a T-test. What's a T-test? statisticshowto.datasciencecentral.com/probability-an…

Tells you how much difference there is between different groups.

So which test when ->

@GRCOhio 21/ But where will they get the data?

- Managed care plans
- Unemployment insurance data
- The Marketplace insurance hub
- Vital statistics (mortality)

@GRCOhio 22/ This quote:

"However, substantial enrollment churn can pose a threat to the study in two ways:"

1) Not enough people stay enrolled
2) People will refuse to participate/will be hard to contact

@GRCOhio 23/ They will track the effects by demographics to ensure there isn't a disparity in the application of the waiver.

This is a good thing that we were interested in knowing as many rural counties are currently exempt (creating uneven application based on race).

@GRCOhio 24/ ODM expects about 50% of the enrollees to "churn" in the program by the end of the first year.

@GRCOhio 25/ By the end of the waiver, the state expects only 8% of the enrollees to remain enrolled in the program.

@GRCOhio 26/ The state may "replenish" the sample in order to account for the high rates of attrition. But:

"...if attrition levels are too high over the length of the study it will be difficult to accurately estimate long-term effects."

@GRCOhio 27/ The waiver population will have weights assigned to them to "look" more like the comparison population to gauge effects.

They'll be monitoring this to see if it's working via sensitivity analysis. What's that? en.wikipedia.org/wiki/Sensitivi…

Yes, I used wikipedia.

@GRCOhio 28/ They mapped out the focus group seating arrangement.

@GRCOhio 29/ Some of the limitations the state sees:

- It's not randomized, though they tried to accommodate
- Lot's of churn
- It'll be hard to stay in touch with people
- The state won't know all the exemptions in its admin data (hmm).

@GRCOhio 30/ Now here are all the metrics they will use in the evaluation!!!

@GRCOhio 31/ I have to dig deeper, but here are some of my initial takeaways...

1) Why didn't we see an implementation plan or the monitoring protocol?
2) Which of these data elements will be available publicly and in what format? At what interval? This should be explicitly defined.

@GRCOhio 32/

3) What and how "churn" is documented is key. The relationships between chrun, group, income, etc., are the basis of the examination.
4) Costs are accounted (yay!) but what about local govt? What about the average PMPM? Would like to see this built out.

@GRCOhio 33/

Generally, this is a pretty robust design with deep potential. That said, the most important thing will be the availability of the data, publicly. Some report at the end (which is what this is describing) won't cut it.

@GRCOhio 34/34

🧵 Last thing - I know I can estimate, but why do we not see numbers (volume) and only rates? If 92% of target population disenrolls, how many people are we talking about, really?

That's it.

Read the document. Submit your comments. 🧵