This paper introduces LeTO, a method for learning constrained visuomotor
policy via differentiable trajectory optimization. Our approach uniquely
integrates a differentiable optimization layer into the neural network. By
formulating the optimization layer as a trajectory optimization problem, we
enable the model to end-to-end generate actions in a safe and controlled
fashion without extra modules. Our method allows for the introduction of
constraints information during the training process, thereby balancing the
training objectives of satisfying constraints, smoothing the trajectories, and
minimizing errors with demonstrations. This "gray box" method marries the
optimization-based safety and interpretability with the powerful
representational abilities of neural networks. We quantitatively evaluate LeTO
in simulation and on the real robot. In simulation, LeTO achieves a success
rate comparable to state-of-the-art imitation learning methods, but the
generated trajectories are of less uncertainty, higher quality, and smoother.
In real-world experiments, we deployed LeTO to handle constraints-critical
tasks. The results show the effectiveness of LeTO comparing with
state-of-the-art imitation learning approaches. We release our code at
this https URL

LeTO 通过可微分轨迹优化引入了基于约束的视觉运动策略学习方法， 同时满足约束条件，平滑轨迹并减小误差。

LeTO：学习具有可微轨迹优化的受限视觉运动策略

LeTO: Learning Constrained Visuomotor Policy with Differentiable  Trajectory Optimization

Prior work has successfully incorporated optimization layers as the last
layer in neural networks for various problems, thereby allowing joint learning
and planning in one neural network forward pass. In this work, we identify a
weakness in such a set-up where inputs to the optimization layer lead to
undefined output of the neural network. Such undefined decision outputs can
lead to possible catastrophic outcomes in critical real time applications. We
show that an adversary can cause such failures by forcing rank deficiency on
the matrix fed to the optimization layer which results in the optimization
failing to produce a solution. We provide a defense for the failure cases by
controlling the condition number of the input matrix. We study the problem in
the settings of synthetic data, Jigsaw Sudoku, and in speed planning for
autonomous driving, building on top of prior frameworks in end-to-end learning
and optimization. We show that our proposed defense effectively prevents the
framework from failing with undefined output. Finally, we surface a number of
edge cases which lead to serious bugs in popular equation and optimization
solvers which can be abused as well.

该研究分析了神经网络中嵌入 optimization layers 的弱点，提出了一种防御方法，防范了 undefined output 对关键应用造成的潜在威胁。