Standard dynamics models for continuous control make use of feedforward
computation to predict the conditional distribution of next state and reward
given current state and action using a multivariate Gaussian with a diagonal
covariance structure. This modeling choice assumes that different dimensions of
the next state and reward are conditionally independent given the current state
and action and may be driven by the fact that fully observable physics-based
simulation environments entail deterministic transition dynamics. In this
paper, we challenge this conditional independence assumption and propose a
family of expressive autoregressive dynamics models that generate different
dimensions of the next state and reward sequentially conditioned on previous
dimensions. We demonstrate that autoregressive dynamics models indeed
outperform standard feedforward models in log-likelihood on heldout
transitions. Furthermore, we compare different model-based and model-free
off-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo
datasets, and find that autoregressive dynamics models consistently outperform
all baselines, achieving a new state-of-the-art. Finally, we show that
autoregressive dynamics models are useful for offline policy optimization by
serving as a way to enrich the replay buffer through data augmentation and
improving performance using model-based planning.

提出了一种基于自回归动力学模型的控制模型，分析了其在离线数据集 RL Unplugged 中的表现，证明其在策略优化中具有潜在应用。