It's a good time to look back on the @RealOzSAGE model led by @Globalbiosec of the NSW Roadmap to Freedom from 13 Sept 2021. How did they do?
Model is at:
ozsage.org/media_releases…
1/
Model is at:
ozsage.org/media_releases…
1/
They modeled three scenarios, lifting restrictions partially at 70% vax with no further lifting, lifting restrictions in two stages at 70/80%, and waiting until 80% to lift the same restrictions. Additionally they modeled the effect of enhanced contact tracing in each case.
2/
2/
NSW followed something like Scenario 3 (lifting restrictions at 70/80%, no enhanced contact tracing). Opening happened somewhat earlier than modeled, 11/18 Oct, rather than 18 Oct/5 Nov.
3/
3/
OzSAGE predicted that 34 days after 80% re-opening, NSW ICUs would enter five weeks of "code black" w/626 covid patients + 300 other patients occupying all ICU beds, eventually peaking at 892 covid ICU patients.
4/
4/
It's been 35 days since NSW's 80% re-opening. How are we doing?
Here's the actual hospitalization (red) against the predictions from @RealOzSAGE (purple is scenario followed):
5/
Here's the actual hospitalization (red) against the predictions from @RealOzSAGE (purple is scenario followed):
5/
...and the actual ICU utilization (red) against the predictions from @RealOzSAGE (purple is the scenario followed):
6/
6/
34 days after re-opening @RealOzSAGE predicted ICU utilization of 626 beds.
Observed utilization was 30 beds, more than 20X lower.
The largest difference between scenarios was only a factor of 3.4; the error is 6 times larger than the effects they are trying to model!
7/
Observed utilization was 30 beds, more than 20X lower.
The largest difference between scenarios was only a factor of 3.4; the error is 6 times larger than the effects they are trying to model!
7/
This isn't terribly surprising; I noted at the time of release that the model was poorly calibrated, and the choice of parameters appeared arbitrary and not based on any objective inputs.
8/
https://twitter.com/MichaelSFuhrer/status/1437552058026971142
8/
This doesn't mean that all modeling is garbage. There are some carefully calibrated models that are have become reasonably predictive especially in the short term.
9/
9/
But anyone can cook up a model and write a press release for it. So we do need to consider the track record of the modelers. In this case they failed. So we should approach their future work with caution.
10/
10/
Continuing to platform poor modeling is counter-productive. It erodes trust in science by the public, and can backfire, leading to the public ignoring *every* message from public health experts.
11/
11/
The irony is that cases in Greater Sydney *are* showing the first signs of an increase now. This is worrying, and we should be debating how to respond. We need accurate, trusted, objective advice from experts.
12/12
https://twitter.com/MichaelSFuhrer/status/1462576593876832257
12/12
• • •
Missing some Tweet in this thread? You can try to
force a refresh
