Imitation learning (IL) is notably effective for robotic tasks where directly programming behaviors or defining optimal control costs is challenging. In this work, we address a scenario where the imitator relies solely on observed behavior and cannot make environmental interactions during learning. It does not have additional supplementary datasets beyond the expert's dataset nor any information about the transition dynamics. Unlike state-of-the-art (SOTA) IL methods, this approach tackles the limitations of conventional IL by operating in a more constrained and realistic setting. Our method uses the Markov balance equation and introduces a novel conditional density estimation-based imitation learning framework. It employs conditional normalizing flows for transition dynamics estimation and aims at satisfying a balance equation for the environment. Through a series of numerical experiments on Classic Control and MuJoCo environments, we demonstrate consistently superior empirical performance compared to many SOTA IL algorithms.

本研究针对在无环境交互的情况下，模仿学习所面临的限制与挑战进行了深入探讨。通过采用马尔可夫平衡方程并引入基于条件密度估计的模仿学习框架，提出了一种新颖的方法来满足环境的平衡方程。实验证明，该方法在经典控制和MuJoCo环境中的性能显著优于许多现有的模仿学习算法。

马尔可夫平衡满足度提升严格批量离线模仿学习的性能