Deep neural policies have recently been installed in a diverse range of
settings, from biotechnology to automated financial systems. However, the
utilization of deep neural networks to approximate the value function leads to
concerns on the decision boundary stability, in particular, with regard to the
sensitivity of policy decision making to indiscernible, non-robust features due
to highly non-convex and complex deep neural manifolds. These concerns
constitute an obstruction to understanding the reasoning made by deep neural
policies, and their foundational limitations. Hence, it is crucial to develop
techniques that aim to understand the sensitivities in the learnt
representations of neural network policies. To achieve this we introduce a
theoretically founded method that provides a systematic analysis of the
unstable directions in the deep neural policy decision boundary across both
time and space. Through experiments in the Arcade Learning Environment (ALE),
we demonstrate the effectiveness of our technique for identifying correlated
directions of instability, and for measuring how sample shifts remold the set
of sensitive directions in the neural policy landscape. Most importantly, we
demonstrate that state-of-the-art robust training techniques yield learning of
disjoint unstable directions, with dramatically larger oscillations over time,
when compared to standard training. We believe our results reveal the
fundamental properties of the decision process made by reinforcement learning
policies, and can help in constructing reliable and robust deep neural
policies.

利用神经网络构建的深度神经决策模型在不同领域中得到广泛应用，但深度神经网络在近似值函数时可能会导致决策边界的稳定性问题，特别是对于高度非凸和复杂的神经网络结构，不可识别、非鲁棒特征对策略决策有较大敏感性。因此，有必要开发能够理解神经网络策略学习中敏感性的技术。本研究介绍了一种理论基础的方法，系统分析了深度神经决策边界在时间和空间上的不稳定方向。通过实验在 Arcade Learning Environment (ALE) 中，我们展示了该方法在识别相关不稳定方向以及测量样本偏移对神经网络策略学习中敏感方向集合的影响方面的有效性。最重要的是，我们证明了最先进的鲁棒训练技术相对于标准训练方法在学习不相交的不稳定方向时，随着时间的推移产生更大的振荡。我们相信这些结果揭示了强化学习策略的决策过程的基本属性，并能帮助构建可靠而稳健的深度神经策略。

深度强化学习的理解与诊断

Understanding and Diagnosing Deep Reinforcement Learning

2D irregular shape packing is a necessary step to arrange UV patches of a 3D
model within a texture atlas for memory-efficient appearance rendering in
computer graphics. Being a joint, combinatorial decision-making problem
involving all patch positions and orientations, this problem has well-known
NP-hard complexity. Prior solutions either assume a heuristic packing order or
modify the upstream mesh cut and UV mapping to simplify the problem, which
either limits the packing ratio or incurs robustness or generality issues.
Instead, we introduce a learning-assisted 2D irregular shape packing method
that achieves a high packing quality with minimal requirements from the input.
Our method iteratively selects and groups subsets of UV patches into
near-rectangular super patches, essentially reducing the problem to
bin-packing, based on which a joint optimization is employed to further improve
the packing ratio. In order to efficiently deal with large problem instances
with hundreds of patches, we train deep neural policies to predict nearly
rectangular patch subsets and determine their relative poses, leading to linear
time scaling with the number of patches. We demonstrate the effectiveness of
our method on three datasets for UV packing, where our method achieves a higher
packing ratio over several widely used baselines with competitive computational
speed.

用于计算机图形学中，我们介绍了一种学习辅助的二维不规则形状装箱方法，通过选择和分组 UV 贴片的子集来实现高装箱质量，并利用深度神经策略线性时间扩展以处理大规模的问题实例。