Why imperfect climate models are more helpful than you think » Yale Climate Connections

Our knowledge of future climate change is largely dependent on complex computer simulation models. These models are huge – they are made up of more than 1 million lines of computer code representing knowledge from dozens of scientific subfields – and no individual scientist fully understands all of a model’s inner workings.

What’s more, rather than focusing on just one model, climate scientists develop and compare dozens or more of such models in various climate model intercomparison projects. The opportunity multiplies but so does the complexity. As a philosopher of science, I am fascinated with the way model comparisons help scientists pull real insights out of this tangle of complexity and uncertainty.

Inspired by a recent Eye on the Storm post written by climate modeler Ricky Rood, here I reflect on how models designed for one purpose often end up generating insight in unexpected ways.

Climate scientists regularly proclaim, citing statistician George Box, that “all models are wrong but some are useful.” But what exactly is meant by “wrong?” On the one hand, it could have to do with the idealizations and abstractions included in all scientific models, including climate models:

• Idealizations involve approximating the truth – for example, climate models parameterize cloud microphysics, which means they represent the larger-scale effects of cloud formation and other processes without representing these processes explicitly.
• Abstractions have more to do with leaving out a process or feature altogether – for example, many of today’s climate models omit the representation of brown carbon aerosols (but they do represent other types of aerosols).

So “All models are wrong” could just mean that all models include some idealizations or abstractions. And just as Box says, models that employ such idealizations and abstractions can still be advantageous. Indeed, a highly idealized model that leaves out tons of real-world processes will be much easier to understand than a model that tries to capture, say, every gust of wind and every drop of rain.

On the other hand, climate models can be wrong in that they give us inaccurate output about past, present, or potential future climates. That is, climate models can give us the wrong answers.

It is easiest to see this for cases of past and present climate, where model output can be compared against observations. These comparisons can be done for a huge variety of climate variables involving averages, variability, and trends of observations like temperature, precipitation, pressure, and other climate phenomena of interest.

A major challenge is that performance analyses of climate models yield a sort of mixed bag: Some models perform better with respect to some variables while others excel elsewhere. For example, if we rank models based on which one does best at simulating global mean surface temperature in the 2010s and which one does best at simulating the rate of Arctic sea ice melt over the past 30 years, different models may well come out on top.

To capture this second idea, then, “All models are wrong” could mean that all models produce at least some inaccurate output. How might such inaccurate models be useful? Here, the story is not so simple.

One answer is to emphasize that models are fit for purpose. As Rood put it, models must be “designed, built, and evaluated to address a specific application.” If your goal is to teach students about the fundamentals of Earth’s energy balance, then a simple model that you can solve by hand may be most apt. If your goal is to project global mean climate at the end of the century under the assumption of slow-to-no emissions reductions, then a suite of state-of-the-art general circulation models would be more suitable.

One challenge in making models “fit for purpose” is that while some priorities are obvious, many of the more fine-grained decisions may not be so clear-cut. For example, while it may seem reasonable to update a model’s cloud representations to better capture cloud feedbacks and thereby improve the model’s ability to simulate temperature changes as a function of carbon dioxide changes, one modeling group that did just this ended up with a model that was less accurate in simulating such temperature changes.

They were able to diagnose why the model became less accurate, but only after the fact and only through iterative testing and analysis conducted over several months. More realistic models are often more complex, and more complex models involve more interactions, feedbacks, and emergent behaviors than scientists can track in their heads. So determining which purpose(s) a model will eventually be fit for before the model has been developed and tested is a fairly daunting task. (Not to mention the fact that model builders and model users are often distinct groups of people, and model builders can’t predict how others will use their model down the road.)

A second challenge is that models that are deemed unfit for some purpose may be co-opted for another (sometimes related) purpose. In such cases, simply judging a model to be “unfit for purpose” would be far too quick. Instead, inaccurate, unfit, or “wrong” models can often be repurposed and ultimately play an essential role in generating new knowledge. Let’s see how this works.

In a 2020 study, Katarzyna Tokarska and colleagues produced a model-based estimate of something called transient climate response. Transient climate response, or TCR, is a measure of climate sensitivity that describes how much global temperatures increase when carbon dioxide concentrations reach a doubling after gradually increasing over some time period. The team’s analysis involved looking at model simulations of recent warming and model simulations of potential future warming, using the latest comprehensive suite of models, called CMIP6, developed in support of the Intergovernmental Panel on Climate Change.

Some of the CMIP6 models did a relatively poor job of simulating recent warming. Let’s call these the too-hot models. The too-hot models are obviously getting something important wrong and have been rightly critiqued on these grounds. However, scientists don’t want to rely on only a small number of models in case those models turn out to be biased or unrepresentative in ways that aren’t yet understood. There is strength in numbers, so to speak.

So instead of throwing out the too-hot models, the team repurposed them to play a key role. These models were plotted alongside the others, and a clear pattern emerged: The more warming a model showed for recent decades, the higher its estimate of TCR.

This pattern told scientists two things. First, the ensemble of models, taken as a whole, was doing a good job of capturing a crucial relationship between past and potential future warming. Second, scientists could focus on the models that both matched the observed historical warming and followed the pattern to update their estimate of TCR. As a result, instead of a wide TCR range with low confidence, they produced a narrower and more robust estimate – shaving nearly 1°C off the top end of the model-based range. The too-hot models weren’t directly used in the final estimate, but they played a crucial role: They helped establish the pattern and showed that, collectively, climate models were performing well. (See here for a deeper dive).

This case reflects a general pattern in climate modeling. In 2005, Ben Santer and colleagues published an analysis in which climate models that individually did a poor job in simulating two variables (surface temperature variability and lower-troposphere temperature variability) collectively did a good job in representing the relationship between those two variables. These “wrong” models provided solid enough evidence to warrant discarding some controversial satellite data. More recently, models that perform worse than others for some purposes of interest may nonetheless be used to provide the test bed data for the “better” models in model weighting frameworks.

In this way, climate modeling research exemplifies what former options trader Nassim Taleb calls “antifragility.” Taleb’s idea is that it’s good to set things up (whether it be your career, a business, your investments, a house, a research project, or whatever) such that you can benefit from errors, failure, or unforeseen disturbances. Something fragile will break due to a certain disturbance. Something antifragile will prosper from said disturbance.

“You want to be the fire and wish for the wind,” Taleb says.

A gust of wind can extinguish a candle, but air – which contains oxygen – amplifies the strength of a wildfire. It’s best to be the wildfire.

So perhaps climate modeling is like the wildfire in an important sense. Finding out that your model performs poorly is a sort of failure, but one that the modeling community is structured to learn from. These failures – whether in simulating Arctic sea ice trends or transient climate response – do not mark the end of usefulness.

On the contrary, as we’ve seen, scientists can repurpose their models to produce new knowledge despite model imperfections. We might be tempted to view this type of activity as a pragmatic workaround. However, I think the value added by flawed models reveals something deeper about how knowledge grows in climate modeling. The models themselves capture many robust relationships even when they err. Through comparison, iteration, and critical interpretation, scientists can make progress because of, not merely in spite of, imperfection.

Ryan O’Loughlin is an assistant professor who researches and teaches philosophy and climate change at Queens College and the CUNY Graduate Center.