Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.

提出了EARLY (Episodic Active Learning from demonstration querY)算法，通过在基于轨迹的特征空间中生成优化的专家演示查询，改善了学习过程中的人类教学体验和学习性能。在三个逐渐增加的导航任务中验证了方法的有效性，结果表明，当演示由模拟的专家策略生成时，我们的方法在所有三个任务中达到了专家水平的性能，收敛速度比其他基准方法快30%以上。随后的用户研究结果证实，在人类专家演示者的情况下，我们的方法仍然可以保持显著更好的收敛性能，同时在任务负荷感知和消耗的人类时间方面实现了更好的用户体验。

从示范中进行情节主动强化学习：给我一个像这样的例子