Transformers have demonstrated remarkable success in natural language
processing; however, their potential remains mostly unexplored for problems
arising in dynamical systems. In this work, we investigate the optimal output
estimation problem using transformers, which generate output predictions using
all the past ones. We train the transformer using various systems drawn from a
prior distribution and then evaluate its performance on previously unseen
systems from the same distribution. As a result, the obtained transformer acts
like a prediction algorithm that learns in-context and quickly adapts to and
predicts well for different systems - thus we call it meta-output-predictor
(MOP). MOP matches the performance of the optimal output estimator, based on
Kalman filter, for most linear dynamical systems even though it does not have
access to a model. We observe via extensive numerical experiments that MOP also
performs well in challenging scenarios with non-i.i.d. noise, time-varying
dynamics, and nonlinear dynamics like a quadrotor system with unknown
parameters. To further support this observation, in the second part of the
paper, we provide statistical guarantees on the performance of MOP and quantify
the required amount of training to achieve a desired excess risk during
test-time. Finally, we point out some limitations of MOP by identifying two
classes of problems MOP fails to perform well, highlighting the need for
caution when using transformers for control and estimation.

利用 transformer 进行最优输出估计问题的研究，提出一种称为 meta-output-predictor（MOP）的模型，该模型在动态系统中表现出类似于卡尔曼滤波的最优输出估计器的性能，并在具有非独立同分布噪声、时变动力学和未知参数的非线性动力学的复杂场景中表现出色，同时指出在控制和估计问题中使用 transformer 时需要谨慎的限制。