Imitation learning is a class of promising policy learning algorithms that is
free from many practical issues with reinforcement learning, such as the reward
design issue and the exploration hardness. However, the current imitation
algorithm struggles to achieve both high performance and high in-environment
sample efficiency simultaneously. Behavioral Cloning (BC) does not need
in-environment interactions, but it suffers from the covariate shift problem
which harms its performance. Adversarial Imitation Learning (AIL) turns
imitation learning into a distribution matching problem. It can achieve better
performance on some tasks but it requires a large number of in-environment
interactions. Inspired by the recent success of EfficientZero in RL, we propose
EfficientImitate (EI), a planning-based imitation learning method that can
achieve high in-environment sample efficiency and performance simultaneously.
Our algorithmic contribution in this paper is two-fold. First, we extend AIL
into the MCTS-based RL. Second, we show the seemingly incompatible two classes
of imitation algorithms (BC and AIL) can be naturally unified under our
framework, enjoying the benefits of both. We benchmark our method not only on
the state-based DeepMind Control Suite, but also on the image version which
many previous works find highly challenging. Experimental results show that EI
achieves state-of-the-art results in performance and sample efficiency. EI
shows over 4x gain in performance in the limited sample setting on state-based
and image-based tasks and can solve challenging problems like Humanoid, where
previous methods fail with small amount of interactions. Our code is available
at this https URL.

提出了 EfficientImitate 这一基于规划的模仿学习方法，成功地将两类看似不兼容的模仿算法：行为克隆和对抗模仿学习，自然地统一到了一个框架中，实现了在性能和样本效率方面的高水平。