Aligning robot behavior with human preferences is crucial for deploying
embodied AI agents in human-centered environments. A promising solution is
interactive imitation learning from human intervention, where a human expert
observes the policy's execution and provides interventions as feedback.
However, existing methods often fail to utilize the prior policy efficiently to
facilitate learning, thus hindering sample efficiency. In this work, we
introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning),
designed for sample-efficient alignment from human intervention. Instead of
inferring the complete human behavior characteristics, MEReQ infers a residual
reward function that captures the discrepancy between the human expert's and
the prior policy's underlying reward functions. It then employs Residual
Q-Learning (RQL) to align the policy with human preferences using this residual
reward function. Extensive evaluations on simulated and real-world tasks
demonstrate that MEReQ achieves sample-efficient policy alignment from human
intervention.

利用 MEReQ（Maximum-Entropy Residual-Q Inverse Reinforcement Learning） 方法，可以通过人类介入进行样本高效的策略对齐。