Since the weather is chaotic, forecasts aim to predict the distribution of
future states rather than make a single prediction. Recently, multiple data
driven weather models have emerged claiming breakthroughs in skill. However,
these have mostly been benchmarked using deterministic skill scores, and little
is known about their probabilistic skill. Unfortunately, it is hard to fairly
compare AI weather models in a probabilistic sense, since variations in choice
of ensemble initialization, definition of state, and noise injection
methodology become confounding. Moreover, even obtaining ensemble forecast
baselines is a substantial engineering challenge given the data volumes
involved. We sidestep both problems by applying a decades-old idea -- lagged
ensembles -- whereby an ensemble can be constructed from a moderately-sized
library of deterministic forecasts. This allows the first parameter-free
intercomparison of leading AI weather models' probabilistic skill against an
operational baseline. The results reveal that two leading AI weather models,
i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though
the former outperforms the latter in deterministic scoring. We also reveal how
multiple time-step loss functions, which many data-driven weather models have
employed, are counter-productive: they improve deterministic metrics at the
cost of increased dissipation, deteriorating probabilistic skill. This is
confirmed through ablations applied to a spherical Fourier Neural Operator
(SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating
effective resolution reveal it has a useful effect on ensemble dispersion
relevant to achieving good ensemble calibration. We hope these and forthcoming
insights from lagged ensembles can help guide the development of AI weather
forecasts and have thus shared the diagnostic code.

天气模型的概率技能与延迟集合相比较，发现 GraphCast 和 Pangu 是领先的 AI 天气模型，在确定性评分方面前者优于后者，但在概率 CRPS 指标上两者得分相同。

一种实用的用于人工智能天气模型的概率基准

A Practical Probabilistic Benchmark for AI Weather Models

A key missing ability of current language models (LMs) is grounding to
real-world environments. Most existing work for grounded language understanding
uses LMs to directly generate plans that can be executed in the environment to
achieve the desired effects. It casts the burden of ensuring grammaticality,
faithfulness, and controllability all on the LMs. We propose Pangu, a generic
framework for grounded language understanding that capitalizes on the
discriminative ability of LMs instead of their generative ability. Pangu
consists of a symbolic agent and a neural LM working in a concerted fashion:
the agent explores the environment to incrementally construct valid candidate
plans, and the LM evaluates the plausibility of the candidate plans to guide
the search process. A case study on the challenging problem of knowledge base
question answering (KBQA), which features a massive environment, demonstrates
the remarkable effectiveness and flexibility of Pangu: A BERT-base LM is
sufficient for achieving a new state of the art on standard KBQA datasets, and
larger LMs further improve the performance by a large margin. Pangu also
enables, for the first time, effective few-shot in-context learning for KBQA
with large LMs such as Codex.

Pangu 是一个通用的基于神经网络的 grounded language understanding 框架，它采用了 LM 的较强区分能力作为建模依据，并且在基于 KBQA 的案例中展示了卓越的性能，同时使得大型 LM 包括 Codex 可以进行有效的 few-shot in-context learning。