Behavioural cloning has been extensively used to train agents and is
recognized as a fast and solid approach to teach general behaviours based on
expert trajectories. Such method follows the supervised learning paradigm and
it strongly depends on the distribution of the data. In our paper, we show how
combining behavioural cloning with human-in-the-loop training solves some of
its flaws and provides an agent task-specific corrections to overcome tricky
situations while speeding up the training time and lowering the required
resources. To do this, we introduce a novel approach that allows an expert to
take control of the agent at any moment during a simulation and provide optimal
solutions to its problematic situations. Our experiments show that this
approach leads to better policies both in terms of quantitative evaluation and
in human-likeliness.

本文介绍了如何将行为克隆与人在环环学习相结合，利用一种新方法在模拟中允许专家随时控制代理并提供最优解，从而解决了行为克隆中的一些缺陷，提高了训练效率和降低了所需资源，实验表明该方法在定量评估和人类相似性方面都具有更好的效果。

人类驱动动态数据集扩充改进行为克隆

Improving Behavioural Cloning with Human-Driven Dynamic Dataset Augmentation

A promising approach for teaching artificial agents to use natural language
involves using human-in-the-loop training. However, recent work suggests that
current machine learning methods are too data inefficient to be trained in this
way from scratch. In this paper, we investigate the relationship between two
categories of learning signals with the ultimate goal of improving sample
efficiency: imitating human language data via supervised learning, and
maximizing reward in a simulated multi-agent environment via self-play (as done
in emergent communication), and introduce the term supervised self-play (S2P)
for algorithms using both of these signals. We find that first training agents
via supervised learning on human data followed by self-play outperforms the
converse, suggesting that it is not beneficial to emerge languages from
scratch. We then empirically investigate various S2P schedules that begin with
supervised learning in two environments: a Lewis signaling game with symbolic
inputs, and an image-based referential game with natural language descriptions.
Lastly, we introduce population based approaches to S2P, which further improves
the performance over single-agent methods.

本研究探讨了一种新的教授人工智能代理人使用自然语言的方法，发现在通过受监督的学习奖励和自我对弈的信号两种方式的训练后比直接从头开始进行自我学习效果更好，并提出了一种基于人群的方法以进一步提高表现。