Some first impressions from the OzSAGE model (ozsage.org/icu-modelling/):
1)Not enough info is given to understand how the model works. They don’t explain how R_eff is calculated, for example, from the restriction settings.
@RichardfromSyd1 @OrinCordus @RageSheen @Globalbiosec
1)Not enough info is given to understand how the model works. They don’t explain how R_eff is calculated, for example, from the restriction settings.
@RichardfromSyd1 @OrinCordus @RageSheen @Globalbiosec
https://twitter.com/RageSheen/status/1437526439553744897
2)They don’t explain their assumptions about ongoing vaccination beyond 80%. This is key to understanding the model and it’s left unstated.
3)They say they adapt a peer-reviewed model, but that model was not used to model time-varying restrictions, and in fact did not include masks or lockdowns at all.
4)They appear to use vaccine effectiveness against infection when they should use vaccine effectiveness against transmission (which is much higher). So it appears that the vaccine effectiveness they use is much lower than Doherty, ANU model, Grattan?
5)Calibration of the model is poor. Doherty at least calibrates all their assumptions on previous observations in NSW and VIC. This model takes only the current outbreak in NSW (cases/day in all NSW) and uses that single time trace to calibrate three parameters of the model...
... (which presumably all affect a single variable R_eff). It appears impossible to deconvolve the three effects, so their relative size in the model appears to be just a guess?
6)Surprisingly, given that they are free to adjust the parameters, they obtain a poor fit of their model to the observed time series. It is also likely to be proven wrong with a week or two! ... 
...If the caseload is much lower than 1890 on 18 October, this will justify treating the model as pessimistic (though in fact it is just means it is poorly calibrated).
7)Assumptions about the post-lockdown conditions appear to be guesses. Guesses relative to the guesses about lockdown conditions. No justification for these settings is given, so in principle they could have put in any numbers to produce whatever outcome was desired.
8)They pick some fairly fine-tuned numbers for the post-lockdown conditions in some cases (46% or 44% movement restrictions, for example). They could have picked anything. In fact, they presumably could have picked settings that caused the peak to diverge much higher...
... It looks to me like they chose parameters to produce results that look “reasonable”. But I’d need to play with their model to know (and they don’t give enough details to do that!)
Summary:
This is not a peer-reviewed model, and the assumptions are not stated with enough transparency that it could be reproduced by others. Some assumptions are clearly arbitrary. It appears rushed, so perhaps some of these criticisms can be addressed in a revised version.
This is not a peer-reviewed model, and the assumptions are not stated with enough transparency that it could be reproduced by others. Some assumptions are clearly arbitrary. It appears rushed, so perhaps some of these criticisms can be addressed in a revised version.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
