In offline Imitation Learning (IL), one of the main challenges is the
\textit{covariate shift} between the expert observations and the actual
distribution encountered by the agent, because it is difficult to determine
what action an agent should take when outside the state distribution of the
expert demonstrations. Recently, the model-free solutions introduce the
supplementary data and identify the latent expert-similar samples to augment
the reliable samples during learning. Model-based solutions build forward
dynamic models with conservatism quantification and then generate additional
trajectories in the neighborhood of expert demonstrations. However, without
reward supervision, these methods are often over-conservative in the
out-of-expert-support regions, because only in states close to expert-observed
states can there be a preferred action enabling policy optimization. To
encourage more exploration on expert-unobserved states, we propose a novel
model-based framework, called offline Imitation Learning with Self-paced
Reverse Augmentation (SRA). Specifically, we build a reverse dynamic model from
the offline demonstrations, which can efficiently generate trajectories leading
to the expert-observed states in a self-paced style. Then, we use the
subsequent reinforcement learning method to learn from the augmented
trajectories and transit from expert-unobserved states to expert-observed
states. This framework not only explores the expert-unobserved states but also
guides maximizing long-term returns on these states, ultimately enabling
generalization beyond the expert data. Empirical results show that our proposal
could effectively mitigate the covariate shift and achieve the state-of-the-art
performance on the offline imitation learning benchmarks. Project website:
https://www.lamda.nju.edu.cn/shaojj/KDD24_SRA/.

提出了一种新的基于模型的框架 —— 离线模仿学习与自适应反向增强（SRA），该框架通过从离线演示构建反向动态模型，以自适应的方式生成导致专家观察到的状态的轨迹，并使用后续的强化学习方法从增强的轨迹中学习，将从专家未观察到的状态转移到专家观察到的状态。实证结果表明，该方法能够有效减轻协变量漖移，并在离线模仿学习基准上达到最先进的性能。

基于模型逆向增强的离线模仿学习

Offline Imitation Learning with Model-based Reverse Augmentation

Offline reinforcement learning (offline RL) is an emerging field that has
recently begun gaining attention across various application domains due to its
ability to learn strategies from earlier collected datasets. Offline RL proved
very successful, paving a path to solving previously intractable real-world
problems, and we aim to generalize this paradigm to a multiplayer-game setting.
To this end, we introduce a problem of offline equilibrium finding (OEF) and
construct multiple types of datasets across a wide range of games using several
established methods. To solve the OEF problem, we design a model-based
framework that can directly apply any online equilibrium finding algorithm to
the OEF setting while making minimal changes. The three most prominent
contemporary online equilibrium finding algorithms are adapted to the context
of OEF, creating three model-based variants: OEF-PSRO and OEF-CFR, which
generalize the widely-used algorithms PSRO and Deep CFR to compute Nash
equilibria (NEs), and OEF-JPSRO, which generalizes the JPSRO to calculate
(Coarse) Correlated equilibria ((C)CEs). We also combine the behavior cloning
policy with the model-based policy to further improve the performance and
provide a theoretical guarantee of the solution quality. Extensive experimental
results demonstrate the superiority of our approach over offline RL algorithms
and the importance of using model-based methods for OEF problems. We hope our
work will contribute to advancing research in large-scale equilibrium finding.

我们提出了基于模型的框架来解决离线均衡点求解（OEF）问题，该框架可直接应用任何在线均衡点查找算法到 OEF 设置中，同时进行最小更改。 OEF-PSRO 和 OEF-CFR 构成了计算 Nash Equilibria（NEs）的两个广泛使用的算法的泛化，而 OEF-JPSRO 计算（粗糙）相关平衡点（（C）CEs）。我们还将行为克隆策略与模型策略相结合，以进一步提高性能并提供解决方案质量的理论保证。广泛的实验结果证明了我们的方法优于离线 RL 算法以及使用模型的方法解决 OEF 问题的重要性。

离线均衡点寻找

Offline Equilibrium Finding

We present a model-based framework for robot locomotion that achieves walking
based on only 4.5 minutes (45,000 control steps) of data collected on a
quadruped robot. To accurately model the robot's dynamics over a long horizon,
we introduce a loss function that tracks the model's prediction over multiple
timesteps. We adapt model predictive control to account for planning latency,
which allows the learned model to be used for real time control. Additionally,
to ensure safe exploration during model learning, we embed prior knowledge of
leg trajectories into the action space. The resulting system achieves fast and
robust locomotion. Unlike model-free methods, which optimize for a particular
task, our planner can use the same learned dynamics for various tasks, simply
by changing the reward function. To the best of our knowledge, our approach is
more than an order of magnitude more sample efficient than current model-free
methods.

本文提出了一种基于模型的机器人运动框架，在只采集 4.5 分钟（45,000 个控制步骤）的四足机器人数据的基础上实现了步行，通过引入跟踪多个时间步长上的模型预测的损失函数来准确地建模机器人的动力学，使学习到的模型可以进行实时控制，此方法比当前的无模型方法在样本效率上提高了一个数量级以上。