Action advising endeavors to leverage supplementary guidance from expert teachers to alleviate the issue of sampling inefficiency in Deep Reinforcement Learning (DRL). Previous agent-specific action advising methods are hindered by imperfections in the agent itself, while agent-agnostic approaches exhibit limited adaptability to the learning agent. In this study, we propose a novel framework called Agent-Aware trAining yet Agent-Agnostic Action Advising (A7) to strike a balance between the two. The underlying concept of A7 revolves around utilizing the similarity of state features as an indicator for soliciting advice. However, unlike prior methodologies, the measurement of state feature similarity is performed by neither the error-prone learning agent nor the agent-agnostic advisor. Instead, we employ a proxy model to extract state features that are both discriminative (adaptive to the agent) and generally applicable (robust to agent noise). Furthermore, we utilize behavior cloning to train a model for reusing advice and introduce an intrinsic reward for the advised samples to incentivize the utilization of expert guidance. Experiments are conducted on the GridWorld, LunarLander, and six prominent scenarios from Atari games. The results demonstrate that A7 significantly accelerates the learning process and surpasses existing methods (both agent-specific and agent-agnostic) by a substantial margin. Our code will be made publicly available.

通过代理感知但对代理不可知的动作建议 (A7) 框架，在深度强化学习中解决了样本效率低下的问题，并通过使用状态特征的相似性作为征求建议的指标，使用代理模型提取有判别性且普遍适用的状态特征，利用行为克隆训练模型以重用建议，以及引入内在奖励来激励利用专家指导，实验结果表明，A7显著加速了学习过程，并在GridWorld、LunarLander和Atari游戏的六个场景中超过了现有方法。

深度强化学习中面向智能体无关行动指导的智能体感知训练