This paper introduces a model-based approach for training feedback
controllers for an autonomous agent operating in a highly nonlinear
environment. We desire the trained policy to ensure that the agent satisfies
specific task objectives, expressed in discrete-time Signal Temporal Logic
(DT-STL). One advantage for reformulation of a task via formal frameworks, like
DT-STL, is that it permits quantitative satisfaction semantics. In other words,
given a trajectory and a DT-STL formula, we can compute the robustness, which
can be interpreted as an approximate signed distance between the trajectory and
the set of trajectories satisfying the formula. We utilize feedback
controllers, and we assume a feed forward neural network for learning these
feedback controllers. We show how this learning problem is similar to training
recurrent neural networks (RNNs), where the number of recurrent units is
proportional to the temporal horizon of the agent's task objectives. This poses
a challenge: RNNs are susceptible to vanishing and exploding gradients, and
na\"{i}ve gradient descent-based strategies to solve long-horizon task
objectives thus suffer from the same problems. To tackle this challenge, we
introduce a novel gradient approximation algorithm based on the idea of dropout
or gradient sampling. We show that, the existing smooth semantics for
robustness are inefficient regarding gradient computation when the
specification becomes complex. To address this challenge, we propose a new
smooth semantics for DT-STL that under-approximates the robustness value and
scales well for backpropagation over a complex specification. We show that our
control synthesis methodology, can be quite helpful for stochastic gradient
descent to converge with less numerical issues, enabling scalable
backpropagation over long time horizons and trajectories over high dimensional
state spaces.

该论文介绍了一种基于模型的方法，用于针对高度非线性环境中的自主代理训练反馈控制器。我们希望该训练策略确保代理满足以离散时间信号时序逻辑（DT-STL）表达的特定任务目标。为了解决长时间跨度任务目标的问题，我们引入了一种基于随机梯度近似算法的梯度近似算法，并提出了适用于复杂规范的 DT-STL 的新的平滑语义。