Behavior Cloning is a popular approach to Imitation Learning, in which a
robot observes an expert supervisor and learns a control policy. However,
behavior cloning suffers from the "compounding error" problem - the policy
errors compound as it deviates from the expert demonstrations and might lead to
catastrophic system failures, limiting its use in safety-critical applications.
On-policy data aggregation methods are able to address this issue at the cost
of rolling out and repeated training of the imitation policy, which can be
tedious and computationally prohibitive. We propose SAFE-GIL, an off-policy
behavior cloning method that guides the expert via adversarial disturbance
during data collection. The algorithm abstracts the imitation error as an
adversarial disturbance in the system dynamics, injects it during data
collection to expose the expert to safety critical states, and collects
corrective actions. Our method biases training to more closely replicate expert
behavior in safety-critical states and allows more variance in less critical
states. We compare our method with several behavior cloning techniques and
DAgger on autonomous navigation and autonomous taxiing tasks and show higher
task success and safety, especially in low data regimes where the likelihood of
error is higher, at a slight drop in the performance.

SAFE-GIL 通过对专家进行对抗性干扰来引导数据采集，从而解决行为克隆方法的 “错误混合问题”，在安全关键状态下更接近专家行为，并在低数据情况下提高任务成功率和安全性。

SAFE-GIL：安全引导的模仿学习

SAFE-GIL: SAFEty Guided Imitation Learning

Even though Deep Reinforcement Learning (DRL) showed outstanding results in
the fields of Robotics and Games, it is still challenging to implement it in
the optimization of industrial processes like wastewater treatment. One of the
challenges is the lack of a simulation environment that will represent the
actual plant as accurately as possible to train DRL policies. Stochasticity and
non-linearity of wastewater treatment data lead to unstable and incorrect
predictions of models over long time horizons. One possible reason for the
models' incorrect simulation behavior can be related to the issue of
compounding error, which is the accumulation of errors throughout the
simulation. The compounding error occurs because the model utilizes its
predictions as inputs at each time step. The error between the actual data and
the prediction accumulates as the simulation continues. We implemented two
methods to improve the trained models for wastewater treatment data, which
resulted in more accurate simulators: 1- Using the model's prediction data as
input in the training step as a tool of correction, and 2- Change in the loss
function to consider the long-term predicted shape (dynamics). The experimental
results showed that implementing these methods can improve the behavior of
simulators in terms of Dynamic Time Warping throughout a year up to 98%
compared to the base model. These improvements demonstrate significant promise
in creating simulators for biological processes that do not need pre-existing
knowledge of the process but instead depend exclusively on time series data
obtained from the system.

在废水处理中，尽管深度强化学习在机器人技术和游戏领域取得了杰出的成果，但在工业流程优化方面仍面临挑战。本研究通过使用模型的预测数据作为输入以进行校正，并改变损失函数以考虑长期预测形态（动态），提供了改进废水处理数据训练模型的两种方法，提高了模拟器在一年中动态时间弯曲方面的性能。这些改进为基于时间序列数据的生物过程模拟器的创建提供了重要的希望。