Carbon Brief: models beat AI on extremes

Carbon Brief published an April 29 analysis saying traditional, physics-based weather models still outperform leading artificial-intelligence systems when forecasters are dealing with record-breaking extremes. The article was based on a new Science Advances paper by Zhongwei Zhang, Erich Fischer, Jakob Zscheischler and Sebastian Engelke that compared the European Centre for Medium-Range Weather Forecasts' High RESolution forecast, or HRES, with AI models including GraphCast, Pangu-Weather and Fuxi. The paper found the numerical model beat the AI systems on record-breaking heat, cold and wind events across nearly all lead times. Carbon Brief said the findings cut against the broader narrative that AI has already overtaken traditional weather models across the board. ### Which models were actually compared? The Science Advances paper compared ECMWF's HRES with five AI systems: GraphCast, GraphCast operational, Pangu-Weather, Pangu-Weather operational and Fuxi. The authors wrote that HRES "still consistently outperforms" those models for record-breaking weather extremes. Carbon Brief said AI models have already surpassed physics-based systems for some forecasting tasks under more typical conditions. (carbonbrief.org) But the outlet said the new analysis focused on a narrower question: how the two model classes perform when weather exceeds previous records. ### What counted as an "extreme" in this study? The authors tested "thousands" of record-breaking hot, cold and windy events recorded in 2018 and 2020, according to Carbon Brief's summary of the paper. (science.org) The Science Advances paper said the study examined record-breaking extremes rather than average forecast skill, which is the benchmark where AI systems have recently posted strong results. (carbonbrief.org) KIT, one of the institutions behind the paper, said the team looked at events that exceeded historical records and found the gap widened as the exceedance became larger. Zhongwei Zhang said AI models generally underestimated the intensity of heat, cold and wind records, and that the underestimation increased as events moved further beyond the range seen in training data. (carbonbrief.org) ### Where did the AI models fall short? The Science Advances paper said forecast errors in the AI models were "consistently larger" than HRES for record-breaking heat, cold and wind across nearly all lead times. It also said the AI systems underestimated both the frequency and intensity of record-breaking events, underpredicted hot records and overestimated cold records as record exceedance grew. (kit.edu) Carbon Brief highlighted the same two points in its write-up: the AI models missed both how often record-breaking events occurred and how strong they were. That matters because the paper framed the problem around extremes that can drive the highest losses for disaster response, infrastructure and public safety. ### Why do the authors say physics-based models still have an edge here? (science.org) Sebastian Engelke told Carbon Brief that AI systems "depend strongly on the training data" and are "relatively constrained to the range of this dataset." KIT's release put the same point more directly, saying neural networks struggle to extrapolate beyond their training domain, while physics-based models are anchored in the laws of physics and can remain reliable when the atmosphere enters states not previously observed. (carbonbrief.org) The paper itself said the findings underscore current limits in AI weather models' ability to extrapolate and to forecast unprecedented extremes. It added that more verification and model development are needed before those systems can be relied on alone for high-stakes uses such as early warning systems and disaster management. ### Does this mean AI is failing at weather forecasting overall? (carbonbrief.org) Carbon Brief said no such blanket conclusion follows from the paper. Its article noted that AI weather models have surpassed traditional systems for some aspects of forecasting and remain attractive because they are faster and use less computing power. KIT said the same comparison holds in ordinary conditions: AI models can be comparable to, or even better than, classical numerical systems on average weather situations. (science.org) The paper's claim is narrower and more specific — that the advantage does not yet hold for record-breaking extremes. ### What comes next for this research? (carbonbrief.org) The Science Advances paper said "further rigorous verification and model development" is needed before AI systems can be used on their own in high-stakes forecasting. Carbon Brief said Engelke called the findings a "warning shot" against replacing traditional models with AI models for weather forecasting too quickly. The paper by Zhang, Fischer, Zscheischler and Engelke is published in Science Advances under DOI 10.1126/sciadv.aec1433. (kit.edu) (carbonbrief.org)