We present a new approach for input optimization of ReLU networks that
explicitly takes into account the effect of changes in activation patterns. We
analyze local optimization steps in both the input space and the space of
activation patterns to propose methods with superior local descent properties.
To accomplish this, we convert the discrete space of activation patterns into
differentiable representations and propose regularization terms that improve
each descent step. Our experiments demonstrate the effectiveness of the
proposed input-optimization methods for improving the state-of-the-art in
various areas, such as adversarial learning, generative modeling, and
reinforcement learning.

我们提出了一种新的 ReLU 网络输入优化方法，明确考虑了激活模式变化的影响。我们分析了输入空间和激活模式空间中的局部优化步骤，提出了具有优越局部下降性质的方法。为了实现这一目标，我们将激活模式的离散空间转化为可微分表示，并提出了改进每个下降步骤的正则化项。我们的实验证明了所提出的输入优化方法在对抗学习、生成建模和强化学习等各个领域中提升了最新技术的有效性。

ReLU 网络输入优化的激活 - 下降正则化

Activation-Descent Regularization for Input Optimization of ReLU  Networks

A major challenge in machine learning is resilience to out-of-distribution
data, that is data that exists outside of the distribution of a model's
training data. Training is often performed using limited, carefully curated
datasets and so when a model is deployed there is often a significant
distribution shift as edge cases and anomalies not included in the training
data are encountered. To address this, we propose the Input Optimisation
Network, an image preprocessing model that learns to optimise input data for a
specific target vision model. In this work we investigate several
out-of-distribution scenarios in the context of semantic segmentation for
autonomous vehicles, comparing an Input Optimisation based solution to existing
approaches of finetuning the target model with augmented training data and an
adversarially trained preprocessing model. We demonstrate that our approach can
enable performance on such data comparable to that of a finetuned model, and
subsequently that a combined approach, whereby an input optimization network is
optimised to target a finetuned model, delivers superior performance to either
method in isolation. Finally, we propose a joint optimisation approach, in
which input optimization network and target model are trained simultaneously,
which we demonstrate achieves significant further performance gains,
particularly in challenging edge-case scenarios. We also demonstrate that our
architecture can be reduced to a relatively compact size without a significant
performance impact, potentially facilitating real time embedded applications.

本文提出一种名为 Input Optimisation Network 的图像预处理模型来学习优化输入数据以适用于特定目标视觉模型，研究了在自动驾驶车辆的语义分割场景中，使用 Input Optimization 的解决方案比使用增强训练数据和对抗性预处理模型更具竞争力，并且结合使用 Input Optimization 网络和微调模型的联合优化方法达到了更好的性能表现。