Reinforcement learning methods typically use Deep Neural Networks to
approximate the value functions and policies underlying a Markov Decision
Process. Unfortunately, DNN-based RL suffers from a lack of explainability of
the resulting policy. In this paper, we instead approximate the policy and
value functions using an optimization problem, taking the form of Quadratic
Programs (QPs). We propose simple tools to promote structures in the QP,
pushing it to resemble a linear MPC scheme. A generic unstructured QP offers
high flexibility for learning, while a QP having the structure of an MPC scheme
promotes the explainability of the resulting policy, additionally provides ways
for its analysis. The tools we propose allow for continuously adjusting the
trade-off between the former and the latter during learning. We illustrate the
workings of our proposed method with the resulting structure using a point-mass
task.

本文采用基于 QP（Quadratic Programs）的方法，取代采用 DNN 的方法来学习 RL 中的价值函数和策略，以此提高其可解释性和简化结构，并且给出了调整可解释性和简化结构的方法。

基于 QP 和 MPC 的强化学习之间的桥梁

Bridging the gap between QP-based and MPC-based RL

This paper presents OptNet, a network architecture that integrates
optimization problems (here, specifically in the form of quadratic programs) as
individual layers in larger end-to-end trainable deep networks. These layers
encode constraints and complex dependencies between the hidden states that
traditional convolutional and fully-connected layers often cannot capture. We
explore the foundations for such an architecture: we show how techniques from
sensitivity analysis, bilevel optimization, and implicit differentiation can be
used to exactly differentiate through these layers and with respect to layer
parameters; we develop a highly efficient solver for these layers that exploits
fast GPU-based batch solves within a primal-dual interior point method, and
which provides backpropagation gradients with virtually no additional cost on
top of the solve; and we highlight the application of these approaches in
several problems. In one notable example, the method is learns to play
mini-Sudoku (4x4) given just input and output games, with no a-priori
information about the rules of the game; this highlights the ability of OptNet
to learn hard constraints better than other neural architectures.

OptNet 是一种深度神经网络体系结构，可以将优化问题作为单独的层集成到大的可端到端可训练深度网络中，并且可以使用灵敏度分析、双层优化和隐式差分等技术来确切地区分这些层，而且能够学会硬约束优化问题，比其他神经网络工具效果更好。