The forecast accuracy of deep-learning-based weather prediction models is
improving rapidly, leading many to speak of a "second revolution in weather
forecasting". With numerous methods being developed, and limited physical
guarantees offered by deep-learning models, there is a critical need for
comprehensive evaluation of these emerging techniques. While this need has been
partly fulfilled by benchmark datasets, they provide little information on rare
and impactful extreme events, or on compound impact metrics, for which model
accuracy might degrade due to misrepresented dependencies between variables. To
address these issues, we compare deep-learning weather prediction models
(GraphCast, PanguWeather, FourCastNet) and ECMWF's high-resolution forecast
(HRES) system in three case studies: the 2021 Pacific Northwest heatwave, the
2023 South Asian humid heatwave, and the North American winter storm in 2021.
We find evidence that machine learning (ML) weather prediction models can
locally achieve similar accuracy to HRES on record-shattering events such as
the 2021 Pacific Northwest heatwave and even forecast the compound 2021 North
American winter storm substantially better. However, extrapolating to extreme
conditions may impact machine learning models more severely than HRES, as
evidenced by the comparable or superior spatially- and temporally-aggregated
forecast accuracy of HRES for the two heatwaves studied. The ML forecasts also
lack variables required to assess the health risks of events such as the 2023
South Asian humid heatwave. Generally, case-study-driven, impact-centric
evaluation can complement existing research, increase public trust, and aid in
developing reliable ML weather prediction models.

深度学习天气预测模型的预测准确性正在迅速提高，但仍存在需求，需要综合评估这些新兴技术的全面性。通过对比三个案例研究，我们发现机器学习的天气预测模型在创纪录的极端天气事件中可能能够达到与高分辨率天气预测系统相似的准确性，但其在极端条件下的外推能力可能比后者更受影响。这些研究可以为现有研究提供补充，增加公众信任，并有助于开发可靠的机器学习天气预测模型。