South Africa’s epidemiological models have proven to be wildly inaccurate. A new commentary by international modellers explains why Covid-19 models are a poor basis for decision-making, while South Africa’s own modellers try, but fail, to explain themselves.

President Cyril Ramaphosa and the secretive politburo in charge of lockdown routinely refer to model projections to assert the necessity of a range of draconian interventions to curtail the rights and liberties of citizens and severely restrict economic activity.

At the start of the lockdown, we were told to expect at least 88 000 and perhaps as many as 350 000 deaths. Later, the lower bound number was tempered to between 40 000 and 48 000 deaths, but even this does not appear to match the reality (of 6 769 deaths to date).

The most notable critic of the government’s various modelling efforts has been the Pandemics Data & Analytics group (Panda), which opened the betting with a paper that estimated the number of life years that will be lost to the lockdown to be 29 times as high as the number of life years lost to Covid-19.

This suggests that the world’s harshest and longest lockdown, which continues to have catastrophic consequences for people’s livelihoods, was an unjustified and absurdly disproportionate response in a country that could ill afford it.

They went on to publish a number of articles reviewing the evidence and concluding that the models continue to ‘grossly overestimate’ deaths.

As I pointed out almost three months ago, the problems with models are not unique to South Africa, and follow a pattern of emphasising worst-case scenarios based on uncertain assumptions, dubious code, and poor data.

At the time, I criticised the secrecy in which South Africa’s modelling exercises were shrouded. The models on which these projections were based were never disclosed for public peer review.

Secret models now published

A number of local modellers from the South African Covid-19 Modelling Consortium (SACMC) last week published an apologia for their work in modelling the course of the Covid-19 pandemic in South Africa.

In it, they claim that ‘we have aimed at making all model outputs and code available for public scrutiny from the beginning’.

They might have aimed at it, but they didn’t actually do it. As they admit: ‘…we are planning to make the model code public for additional scrutiny’.

They appear finally to have done so, on the quiet, on Friday 24 July. It took a Promotion of Access to Information Act request from Panda to elicit a website address where the model code and data for SACMC’s National COVID-19 Epi Model (NCEM) could be found.

The earlier models, created by the South African Centre for Epidemiological Modelling and Analysis (SACEMA) have been abandoned, and were never published.

Now that the current models are finally public, I urge anyone with the requisite coding, statistics, modelling or epidemiology background, to review them in detail and publish their findings. Panda surely will, but the more reviewers, the better.

A modelling manifesto

For now, let’s have a closer look at the SACMC modellers’ apologia.

At the outset, they recognise – and concur with – a new commentary by a larger group of international modellers, published in the respected journal Nature, entitled Five ways to ensure that models serve society: a manifesto.

That paper points out many inherent shortcomings of computer models. It is worth quoting at some length:

‘…computer modelling is in the limelight, with politicians presenting their policies as dictated by “science”. Yet there is no substantial aspect of this pandemic for which any researcher can currently provide precise, reliable numbers. Known unknowns include the prevalence and fatality and reproduction rates of the virus in populations. There are few estimates of the number of asymptomatic infections, and they are highly variable. We know even less about the seasonality of infections and how immunity works, not to mention the impact of social-distancing interventions in diverse, complex societies.

Mathematical models produce highly uncertain numbers that predict future infections, hospitalizations and deaths under various scenarios. Rather than using models to inform their understanding, political rivals often brandish them to support predetermined agendas. To make sure predictions do not become adjuncts to a political cause, modellers, decision makers and citizens need to establish new social norms. Modellers must not be permitted to project more certainty than their models deserve; and politicians must not be allowed to offload accountability to models of their choosing.’ (Emphasis mine.)

It goes on to list five ‘best practices’ for quality modelling, summarised here:

  1. Mind the assumptions. Make sure assumptions are reasonable, perform uncertainty and sensitivity analyses on them, and make it clear what the compounded uncertainties mean for the final output.
  2. Mind the hubris. It is tempting to increase a model’s complexity to better capture reality, but each added element introduces new uncertainty, so increasing complexity can lead to an ‘uncertainty cascade’, making the results less, rather than more, accurate.
  3. Mind the framing. A range of factors, from modellers’ own biases, interests and disciplines to the choice of tools can influence, and even determine, the outcome of the analysis. Models are easily manipulated to produce desired results. To put it in jargon, the authors write, ‘qualitative descriptions of multiple reasonable sets of assumptions can be as important in improving insight in decision makers as the delivery of quantitative results’.
  4. Mind the consequences. Excessive focus on numbers ‘can push a discipline away from being roughly right towards being precisely wrong’. In the case of Covid-19, the issue is not only projected deaths, or projected need for hospital beds, but also projected unemployment, projected economic costs, projected non-Covid-related deaths, and projected consequences for civil liberties.
  5. Mind the unknowns. When there is no clear or simple answer to a question, modellers should say so. Failure to acknowledge ignorance not only leads to wrong policy options, but also allows politicians to abdicate responsibility for their actions by saying ‘we only acted upon what the models told us’.

Ultimately, the paper concludes: ‘Mathematical models are a great way to explore questions. They are also a dangerous way to assert answers. Asking models for certainty or consensus is more a sign of the difficulties in making controversial decisions than it is a solution. … We are calling not for an end to quantification, nor for apolitical models, but for full and frank disclosure. Following these five points will help to preserve mathematical modelling as a valuable tool. Each contributes to the overarching goal of billboarding the strengths and limits of model outputs. Ignore the five, and model predictions become Trojan horses for unstated interests and values. Model responsibly.’

The modellers’ apologia

The SACMC modellers no doubt would like to think that they have been fully cognisant of these issues, and have clearly communicated the limitations of their models to their political clients. However, the explanations of their own results raises more questions than answers.

If, as the local modellers concede, there is significant uncertainty regarding almost all central aspects of how the virus behaves, including its prevalence, reproduction rate, the proportion of infected people who remain asymptomatic, the role of seasonality, whether cross immunity to the virus from other infections exists, whether immunity to the virus itself persists, and the impact of non-pharmaceutical interventions such as physical distancing and mask wearing, on what basis can we have any confidence at all in the model outputs?

So-called Susceptible-Exposed-Infectious-Recovered (SEIR) models have failed miserably in countries that were well ahead of us on the infection curve. Why then, do the SACMC modellers continue to rely on such a model, and why, given that they have historical data from those countries, is there still ‘considerable uncertainty regarding the likely trajectory of the Covid-19 epidemic in South Africa overall, and perhaps more so due to recent flattening of the growth in cases in the Western Cape’?

They claim to have incorporated international experience into the models, but their models would have failed in countries that are ahead of South Africa on the curve, so should be expected to fail in South Africa, too.

The modellers say that they ‘present [their] estimates with uncertainty bands reflective of variation in the parameters driving the model and the model process itself’. Yet even with very wide uncertainty bands, some of the modelled projections failed on the low side, within a month of publication.

Best effort rather poor

They chose one long-term projection and one short-term projection to demonstrate how well they did. Both estimated the death count on 1 July.

The long-term projection, published on 6 May, ranged from an optimistic scenario of 822 deaths (between 431 and 1 618) to a pessimistic scenario of 5 486 deaths (between 2 849 and 9 869).

To say ‘somewhere between 400 and 10 000-odd deaths’ is not a projection. It’s a guess, hedged to meaninglessness. Even ‘between 822 and 5 486 deaths’ is not a useful projection. That the actual death toll of 2 952 on 1 July falls in that massive range should hardly be a point of pride. Any undergraduate with a basic curve-fitting technique could have given you a similar answer.

Even the short-term projection, published on 12 June, had a massive error range. The estimate was 3 810 deaths (between 1 880 and 7 270), which reflects a 50% error bar on the downside, and a 90% error bar on the upside. The actual death toll does fall within this spectacularly wide range, as one would expect, but the projection still exceeded reality by a massive 30%.

Could you plan a party, knowing that you’d have to cater for between 18 and 72 people? Or worse, between four and 100 people? That’s how useful the SACMC model projections have been. Now imagine you’re the manager of a cash-strapped, understaffed hospital, facing these sort of numbers and being expected to plan for them.

And this is their best effort, which they chose to highlight. That the models repeatedly failed on the downside is never explained.

Western Cape breaks models

The modellers observe that deaths in the Western Cape have plateaued for four weeks, and hospital admissions in the province peaked on 22 June. Their models did not predict this. In fact, they got the Western Cape’s numbers spectacularly wrong. The death count was lower than their most optimistic projection. They over-estimated the number of ICU beds that would be needed by between 12 and 16.5 times. The most optimistic projection for total beds needed was more than double the actual number at the peak.

They have no explanation for the plateau in the Western Cape, although Panda’s own very basic model, based on nothing but publicly available data, predicted it far more accurately.

Worse, the modellers wonder whether they over-estimated the Covid-19 deaths in the Western Cape, and conclude that they don’t know. How much more is there to know, except to note that actual deaths were lower than their lower bound, and their models failed to anticipate the peak? Clearly, they did over-estimate. There’s no uncertainty about that.

Although the Western Cape numbers surprised them, the modellers claim they cannot infer anything from the change, or say what it means for the rest of the country. It’s almost as if real-world data is an inconvenience to them, rather than a critical input into their models.

Trojan horses

They say that ‘individual behaviour changes are unpredictable and difficult to model, [so] these are not included in our models’. A few paragraphs later, they say that they regularly update ‘our assumptions about the adherence of the population to the restrictions’.

Which is it? Do they model behaviour changes, or don’t they? If not, how can they justify their failure to model one of the single biggest factors in the spread of the disease? And if they do, where do they get the data on these behaviour changes?

They claim they’ve had to decrease their assumptions about how fast the virus spreads, because there turned out to be fewer cases than expected. But they have no idea why this happened. They speculate, based on no evidence at all, that it might have to do with adherence to restrictions. But that would conflict with the evidence that the Western Cape’s numbers have started to decline while restrictions were being lifted.

The SACMC modellers promise that they’ll continue to update their numbers, and ‘continue to strive towards making our models as useful as possible’. However, it is clear that their models are not very useful at all. Merely updating the numbers periodically will not fix that.

They are exactly what the Nature paper warns us against: ‘Trojan horses for unstated interests and values’. And we all know what those interests and values are.

[Picture: Martin Sanchez on Unsplash]

The views of the writer are not necessarily the views of the Daily Friend or the IRR

If you like what you have just read, subscribe to the Daily Friend

22 COMMENTS

  1. Once again, Bravo Ivo, your rational mind versus worshippers at the shrine of the computer screen. Ivo 10, Screen worshippers 0.
    BTW everyting said in this remarkable piece of sanity applies to the climate scare. It should be required reading for every wannabee journalist, and every sociology student at every university. Otherwise I fear it will be like preaching to the choir.

  2. There is reason to believe in the efficacy of the COVID COMMAND COUNCIL’s diktats to prevent infection of influenza killing less people than the same period last year. Clearly wearing schnozzle-sacks and enforced loss of contact with relatives, neighbours and strangers – total alienation in fact while under solitary confinement house arrest – is indeed flattening the “curve” – not to mention the economy. Although that was flat-lining prior to the plandemic. In a bid to hasten The End – and the arrival of the New World Order – there is the possibility that the CCC will now consider a further weapon in this fight against ‘flue…..to the wearing of nose-bags and Social Distancing (neither of which seems really satisfactory) we may soon be required to walk backwards when emerging (under strict conditions) from our homes. Thus any infected droplets will dissipate in our wakes. Of course this might predictably create some increased load upon the medical infrastructure but since the total prohibition on alcohol has lessened it, walking backwards may now be the way to go.

    • I concur with you 100% Roger, but you left out one little gem that I saw recently – eye protection. The insanity knows no bounds. And sadly there are still people who believe the fear-mongering and that masks prevent transmission. There is no hope for them. They must take their deadly CV vaccine and take the consequences. There is still more to come – the “COVI-PASS”. 15 countries already signed up, including South Africa. You will need a blood test to confirm your “immune status” before you can fly internationally again. And that is obviously just the beginning. When will the media wake up to this properly? The MSM won’t. I hope that the Daily Friend will and that they drive home the tyrannical future that is ours if we don’t do anything constructive about it.

  3. Thank you, Ivo. Now apply the same reasoning to climate modelling which parades as “science”.

    Climate “science” has become a religion with its underlying beliefs based on cherry picking data that would put Erich Von Daniken with his Charriot of the Gods to shame. No falsifiability (a la science philopher Karl Popper), no respect for known unknowns, full of biasses including the lust for “research” moneys.

    • Quite correct JS. The MO is exactly the same.

      A tool used to justify a political decision with scant regard for society and the consequential effects (Damage- to the extreme). The trade-off is ignored completely.

      This has been the tool used by the UN-IPCC and now the UN-WHO to try and legitimise Government response to their diktats by an unelected organisation of bureaucrats. This is what is being used to legitimise the UN AGENDA 21/30. The very Climate Change Bill and the subsequent Carbon Tax Bill and now the Carbon Currency implementation and the Cashless society is being driven by this con trick.

      People need to wake up and the political parties either need to fight to stop this and take this matter to the constitutional court.

      This is imperative. I urge everyone to look at this U-Tube video by
      Agenda 21 Truth – South Africa- https://youtu.be/2lmm0rG2Ga0

  4. Fantastic commentary Ivo Vegter. I entirely agree with your views and me and my family as professional people also used local data and we predicted no more detahs than 15000 for entire South Africa by end of the year. It is absolutely shocking how badly the entire SACMC have made predictions based on models that has no rationale. I suppose the livelihoods lost because of this is too large to even think about.

    Please keep on writing these extremely intelligent and useful comments ….may be in the end sense will prevail.

  5. Very good Mr. Vegter. 83% of the covid cases in the Western Cape have already recovered and .04% of this group have died if one can believe the Western Cape’s figures. . This means the trajectory is going down and the projected death rate nationwide will be closer to 10 to 20,000 people, not 400,000 as predicted. Maybe my math is flawed. Horrifying. I think it’s about the control of protesters by the army and police. people who do not have an income and have to face their families

  6. The ONLY reason for this LOCKDOWN is to ”experiment” whether the SNOUTER can ”stifle the minority, which is a CRIME AGAINST HUMANITY – the State Of Emergency” in ’85 was only allowed (by parliament) for 21 days not 120 days!!!!!!!

  7. Two rumours I heared: 17 000 unexplained deaths, and Western Cape doctors ordered to test only persons over 55 years old since a few weeks ago. That flattened the curve, I presume, Mr Watson.

    • My wife does Covid-19 testing in the Western Cape, and she certainly does not test only over-55s. I’m also pretty suspicious of that 17,000 number. I expect we’ll get more clarity on it soon.

      • SA Medical research Council weekly mortality report of 22 Jul. Refers.
        Specific doctor at specific Cape Town hospital confirmed 55yr cut off.
        Hope clarity prevails soon.

  8. I usually agree with most of what Ivo writes. The one problem I have here is the assumption that the published numbers for “the actual death toll” are in any way accurate. It is also the only bit of information which Ivo seems to have any trust in.

    Whereas, if you do just a straight comparison between South Africa, the UK and Germany’s reported Covid-related numbers of number of tests per population, number of positives per test, and number of fatalities per case, one can easily conclude that the number of Covid-related fatalities in South Africa is under reported by a factor of as much as 10. Which if true, means that the South African modelers actually got it spot-on.

    I cannot for a moment believe that South Africa’s population is 10x more resistant to Covid deaths than Europe’s, or that South Africa is 10x more effective in the treatment of Covid than Germany. Somewhere, the reported numbers do not make sense. One can hardly blame the modelers for that.

    So, I believe that there is a bit of chauvinism involved here: Why does Ivo favor the published number of Covid-deaths as accurate, whereas everything else as not?

    • It is plausible, and even likely, that some Covid-19 deaths go unreported. However, the models are calibrated to testing data, hospitalisations and hospital-recorded deaths. SA’s case-fatality ratio, accurate or not, is baked into the models. The models are not designed to predict unreported, non-hospital death, and their projections should be consistent with hospital-recorded deaths.

      According to the MRC, excess mortality between 6 May and 14 July was 17 000, of which 4 200 were confirmed Covid-related deaths. So it is entirely implausible that the true Covid-19 death toll is 10x higher than reported, as you suggest.

      Moreover, Panda has pointed out[1] that the MRC excess mortality number has been manipulated, and the true number is closer to half of that, or 8 500. Of those, many, and probably most, are unrelated to Covid-19 itself, but related to the lockdown, which kept many sick people away from healthcare services, and caused hunger and malnutrition to boot.

      So it is implausible that Covid deaths have been under-reported by a large margin.

      It is quite possible that SA’s population is more resistant to Covid-19 than Europe’s people. For a start, it skews much, much younger, and youth is a protective factor. There is also evidence to suggest that genetics may play a role, that TB inoculation (which is rare in Europe but universal in SA) might be protective, that climate makes a difference, and that HIV treatment (also rare in Europe) may be protective. Of these, I think age distribution is the biggest factor, while the influence of the BCG vaccination against TB needs to be studied more.

      South Africa’s case-fatality ratio is not in line with European countries, but it is consistent with that of New Zealand, Russia, Kenya, Zimbabwe, Ethiopia, Argentina, Thailand, and the Diamond Princess cruise ship. There’s nothing particularly remarkable about our Covid-19 death rate.

      So I stand by my conclusion, which is that the model projections grossly exaggerated the reality.

      [1] PANDA | Don’t play around with Covid-19 deaths and statistics https://www.news24.com/news24/Columnists/GuestColumn/panda-dont-play-around-with-covid-19-deaths-and-statistics-20200723

  9. A clear thinking analysis. I would question though whether it is just “plausible” that there are unreported deaths. I think it does not take into account how far the country has deteriorated in recent years. Given the extremely low administrative capability of the civil service, the fact that much of it has been in lockdown, the general poor state of government hospitals and mortuaries and pathology services, overworked doctors, weak home affairs department, failed municipalities etc, I would say the odds are high that deaths are not being reported at all or if they are, they are only recorded months later. Doctors will be signing death certificates without waiting for pathology reports on whether Covid was the cause of death or not, assuming that the victims even get to see a doctor. One might add that in a crisis, the accurate recording of stats is not usually a priority. When crocodiles are biting, one doesn’t worry about the quality of the water!
    I think it will be a long time before realistic figures are known, if ever, and even those will be statistical estimates based on excess deaths and not proven covid related ones.

  10. I fully agree with your article.
    However there is a massive agenda that is very hard to explain and convince people to think about what is happening. It is fobbed off as a conspiracy theory. I cannot urge enough for the South African society and citizens to broadcast the urgency and extreme danger of this, UN- Agenda 2030 poses on us all.

    This is a scam driven under a variety very deceptive noble causes.
    It is manipulation of the highest order.
    This is the UN Agenda of sustainable development. It is pure and simple global communism happening before our very eyes.
    This current con-trick was already tested by the WEF under the Event 201 Pandemic Excercise that was played out in October 2019.
    The next phase is the “Great Reset”

    This Worldwide Covid19 scam is also being driven regionally by the ANC. https://off-guardian.org/2020/01/29/coronavirus-media-reveals-a-very-familiar-agenda/

    The models are simply there a deception for all to squabble about. Its a RED HERRING.

    They are as faulty and the Basic Reproduction Number (R0) models, Delamater, Paul & Street, Erica & Leslie, Timothy & Yang, Y. & Jacobsen, Kathryn. (2019). Complexity of the Basic Reproduction Number (R0). Emerging Infectious Diseases. 25. 1-4. 10.3201/eid2501.171901.

    The testing using PCR Tool has the objective of supporting the model predictions numbers originally created by Niel Ferguson of Imperial London College fame that the UN-WHO used for its agenda promotion.
    The PCR support tools have been condemned by many.
    https://off-guardian.org/2020/06/27/covid19-pcr-tests-are-scientifically-meaningless/
    There was a Fact Check being used by Social media to attempt to debunk this.
    Here is the response by the Authors
    https://off-guardian.org/2020/07/31/open-letter-refuting-politifacts-fact-check/

    Behind this scam is a selected Media campaign who trumpet the same message over and over.
    The exact MO is also being used by the ANC to drive its part of is agreement with the UN. There is common cause.

    The ANC NDP is part of this.
    PLEASE look at the Video- https://youtu.be/2lmm0rG2Ga0

  11. The Lockdown is not to save the citizens of death but the save the ANC of it. In fact what they are doing is signing their own death certificate, they are killing off their own voters through starvation, due to companies closing down and will never be able to lift themselves out of the mess the ANC has caused with this now every overdue lockkdown, There is no more money left due to thousands now jobless, crime on the increase in fact getting pretty dangerous. how must people feed their families? But its ok for those in Government positions they get paid every month, fatcat salaries, eating well. Really if President Rhamaphosa do not open up the cities now, deaths will reach a all high and it will not be the Virus killing them but He will be responsible for their deaths preventing them of earning a living. WAKE UP FOR HEAVEN SAKE

LEAVE A REPLY

Please enter your comment!
Please enter your name here