Scientist Admits All Disease Models Are “Wrong” – Working to Fix

Doctor Presenting Outbreak Model Results

An international team of researchers has developed a new mathematical tool that could help scientists to deliver more accurate predictions of how diseases, including COVID-19, spread through towns and cities around the world.

Rebecca Morrison, an assistant professor of computer science at CU Boulder, led the research. For years, she has run a repair shop of sorts for mathematical models—those strings of equations and assumptions that scientists use to better understand the world around them, from the trajectory of climate change to how chemicals burn up in an explosion.

By definition, models are simplified from reality. In some way or another, all models are wrong.” — Rebecca Morrison

As Morrison put it, “My work starts when models start to fail.”

She and her colleagues recently set their sights on a new challenge: epidemiological models. What can researchers do, in other words, when their forecasts for the spread of infectious diseases don’t match reality?

In a study published on May 4, 2020, in the journal Chaos, Morrison and Brazilian mathematician Americo Cunha turned to the 2016 outbreak of the Zika virus as a test case. They report that a new kind of tool called an “embedded discrepancy operator” might be able to help scientists fix models that fall short of their goals —effectively aligning model results with real-world data.

Zika Virus Mosquitos Aedes aegypti

The Zika virus can be transmitted by mosquitos from the species Aedes aegypti. Credit: CDC

Morrison is quick to point out that her group’s findings are specific to Zika. But the team will soon try to adapt their methods to help researchers to get ahead of a second virus, COVID-19.

“I don’t think this tool is going to solve any epidemiologic crisis on its own,” Morrison said. “But I hope it will be another tool in the arsenal of epidemiologists and modelers moving forward.”

When models fail

The study highlights a common issue that modelers face. 

“There are very few situations where a model perfectly corresponds with reality. By definition, models are simplified from reality,” Morrison said. “In some way or another, all models are wrong.”

Zika Virus Health Advisory

A Zika virus health advisory posted near an airport security checkpoint in San Juan, Puerto Rico, in 2016. Credit: CDC

Cunha, an assistant professor at Rio de Janeiro State University, and his colleagues ran up against that very problem several years ago. They were trying to adapt a common type of disease model—called a Susceptible, Exposed, Infected or Recovered (SEIR) model—to recreate the Zika virus outbreak from start to finish. In 2015 and 2016, this pathogen ran rampant through Brazil and other parts of the world, causing thousands of cases of severe birth defects in infants.

The problem: No matter what the researchers tried, their results didn’t match the recorded number of Zika cases, in some cases miscalculating the number of infected people by tens of thousands. 

Such a shortfall isn’t uncommon, Cunha said.

“The actions you take today will affect the course of the disease,” he said. “But you won’t see the results of that action for a week or even a month. This feedback effect is extremely difficult to capture in a model.” 

Rather than abandon the project, Cunha and Morrison teamed up to see if they could fix the model. Specifically, they asked: If the model wasn’t replicating real-world data, could they use that data to fashion a better model?

Zika Model Comparison

Graphic showing the success of different models in predicting recorded Zika cases in Brazil in the hundreds of thousands over the weeks since the outbreak began. Blue triangles represent the recorded cases, while the orange, green and purple lines show various versions of the original disease model. The red line corresponds to the “enriched” model that Morrison and Cunha developed. Credit: Morrison and Cunha, 2020

Enter the embedded discrepancy operator. You can picture this tool, which Morrison first developed to study the physics of combustion, as a sort of spy that sits within the guts of a model. When researchers feed data to the tool, it sees and responds to the information, then rewrites the model’s underlying equations to better match reality.

“Sometimes, we don’t know the correct equations to use in a model,” Cunha said. “The idea behind this tool is to add a correction to our equations.”

The method worked. After letting their operator do its thing, Morrison and Cunha discovered that they had nearly eliminated the gap between the model’s results and public health records. 

Being honest

The team isn’t stopping at Zika. Morrison and Cunha are already thinking about how they can deploy their same strategy to try to improve models of the coronavirus pandemic. 

Morrison doubts that any disease model will ever be 100% accurate. But, she said, these tools are still invaluable for informing public health decisions—especially if modelers are up front about what their results can or can’t tell you about a disease.

“This epidemic has revealed how difficult it is to model a real system,” Morrison said. “But I hope that people don’t take that to mean that we shouldn’t trust our scientists.”

Reference: “Embedded model discrepancy: A case study of Zika modeling” by Rebecca E. Morrison and Americo Cunha Jr., 4 May 2020, Chaos.
DOI: 10.1063/5.0005204

1 Comment on "Scientist Admits All Disease Models Are “Wrong” – Working to Fix"

  1. “causing thousands of cases of severe birth defects in infants.” A nasty disease yes – BUT no – the many thousands of deformed babies were MOSTLY in a SPECIFIC area of S. America WHERE THEY HAD JUST BEEN SPRAYING CHEMICALS on huge areas of farmland…..way too much of a pattern for that to be a coincidence….very convenient to have a Zika problem that can hide what was really a big CHEMICAL SPRAYING PROBLEM…thereby saving any BIG-CHEM corporations from getting sued ??? I try to read everything about everything, and it quite apparent to me that those with the money WILL try to protect themselves from untoward results of anything they produce ! Nobody could blame anybody if it was a mosquito’s fault could they ?

Leave a comment

Email address is optional. If provided, your email will not be published or shared.