, 24 tweets, 22 min read
My Authors
Read all threads
I want to spend some time on this, because it's a perennial source of friction. Groups like @MalariaAtlas and @IHME_UW that produce annual time series of disease metrics spend a lot of time trying to work through the "why does the past change" criticism.
@MalariaAtlas @IHME_UW Punchline: I believe that the solution is to change communication conventions such that the uncertain nature of modeled results is front and center, all the time.
@MalariaAtlas @IHME_UW Reporting only mean values is definitely misleading. But even the "Mean (Lower CI- Upper CI)" format minimizes the fact that these estimates are more accurately described by ranges than by single points.
@MalariaAtlas @IHME_UW Consider three sentences:
1. We estimate X cases in 2018.
2. We estimate X cases (95% CI A-B) in 2018.
3. We estimate between A and B cases in 2018, with 95% confidence.
Only #3 highlights the most relevant values, while making clear that we *do not actually know* the case count.
@MalariaAtlas @IHME_UW I know this feels strange and clunky, but we encounter this type of language all the time. When was the last time you saw a weather prediction for exactly 5mm of rain? My guess is never, because climate modeling has embedded uncertainty into its language at every level.
@MalariaAtlas @IHME_UW So why should health modeling, which has *dramatically worse* data collection than climate modeling, pretend to be *more certain* about its outcomes?
@MalariaAtlas @IHME_UW I think that modelers, model communicators, and policymakers alike need to make a concerted collaborative push to change the way we talk about model outputs, that will also improve the way we interpret and act on modeled results.
@MalariaAtlas @IHME_UW Ok, I'll get off my soapbox now to say a few words about @ricprice99's main point: why the heck does the whole time series change so much year on year?
@MalariaAtlas @IHME_UW @ricprice99 Note: I did not work on the WMR estimates, so can't speak to them directly. I'm speaking generally about reasons one *might* see changes in modeled time series results.
@MalariaAtlas @IHME_UW @ricprice99 First point: Small changes in rate space lead to big changes in count space. Illustrative example: Say you're working on WMR 2016, and this is your data. You run an analysis and get an incidence time series as represented by the line.
@MalariaAtlas @IHME_UW @ricprice99 Then, two years later, you run the same analysis, now with an extra time point. The slope of the line isn't that different, and your new time series falls well within the confidence interval of your old one.
@MalariaAtlas @IHME_UW @ricprice99 However, when you're working with population denominators as big as the ones we see in malaria, when you convert to case counts, your new estimates will show *millions of extra cases* for past years compared to your previous estimates!
@MalariaAtlas @IHME_UW @ricprice99 (Burden estimation for malaria is of course much more complicated than "drawing a straight line through data", but the basic idea translates.)
@MalariaAtlas @IHME_UW @ricprice99 Second point: Modeling strategies for burden estimation are constantly changing. Population denominators get updated. We get improved covariates. We implement more accurate estimation methods. We find bugs. These are good changes, but they cause trouble.
@MalariaAtlas @IHME_UW @ricprice99 We want to make our results as accurate as we possibly can, but we understand that "better" estimates are not always welcome when billion-dollar policy decisions have just been made off of the old estimates.
@MalariaAtlas @IHME_UW @ricprice99 This isn't a problem with an easy or one-sided solution. It mandates a closer relationship between modelers and stakeholders, and (again) a fundamental shift in the language we use to communicate and act on these estimates.
@MalariaAtlas @IHME_UW @ricprice99 The good news: these types of deep relationships are becoming more common and more expected, most prominently in the HBHI strategy now underway. I'd highlight @SwissTPH and @MORUBKK as groups that have been modeling this good behavior for many years already.
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK I'm also seeing a lot of energy towards efforts to move modeling and data science for malaria into the hands of researchers from endemic countries, where it belongs. For much more detail on that, follow @Fredros_Inc and friends.
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK @Fredros_Inc UPDATE: Wow, just saw this really misleading plot from a FT article that calls for a "no-holds-barred inquiry" into these deviations
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK @Fredros_Inc I took the liberty of making a plot of my own from those same reports, but including the CI's that are listed in the report right next to the mean counts:
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK @Fredros_Inc (Couldn't find the data point for 2000 in the FT article, full link is here on.ft.com/33Z9GzJ). Even preserving a truncated y-axis and using x-axes to exaggerate the time series effect, the second plot clearly shows that the new estimates are well within what we'd expect.
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK @Fredros_Inc I want to be clear here that it is good and proper for people to question modeled results, and our job as modelers to make our work comprehensible. We too have failed if you can't interpret the uncertainty we report.
@MalariaAtlas @IHME_UW @ricprice99 @SwissTPH @MORUBKK @Fredros_Inc But the article borders on accusing WHO of malpractice, which feels unreasonable given that the reports *do* clearly list CI's. Whoever collated the data for the plot they published had to scroll right past the "Lower" and "Upper" reported values in the tables.
As @Fredros_Inc and others have noted, the best part of the FT article is @NoorAbdisalan redirecting our attention where it belongs: towards improving surveillance systems so we can gain a true understanding of how to save lives.
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Amelia Bertozzi-Villa

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!