In this work, we explore the influence of entropy change in deep learning
systems by adding noise to the inputs/latent features. The applications in this
paper focus on deep learning tasks within computer vision, but the proposed
theory can be further applied to other fields. Noise is conventionally viewed
as a harmful perturbation in various deep learning architectures, such as
convolutional neural networks (CNNs) and vision transformers (ViTs), as well as
different learning tasks like image classification and transfer learning.
However, this paper aims to rethink whether the conventional proposition always
holds. We demonstrate that specific noise can boost the performance of various
deep architectures under certain conditions. We theoretically prove the
enhancement gained from positive noise by reducing the task complexity defined
by information entropy and experimentally show the significant performance gain
in large image datasets, such as the ImageNet. Herein, we use the information
entropy to define the complexity of the task. We categorize the noise into two
types, positive noise (PN) and harmful noise (HN), based on whether the noise
can help reduce the complexity of the task. Extensive experiments of CNNs and
ViTs have shown performance improvements by proactively injecting positive
noise, where we achieved an unprecedented top 1 accuracy of over 95% on
ImageNet. Both theoretical analysis and empirical evidence have confirmed that
the presence of positive noise can benefit the learning process, while the
traditionally perceived harmful noise indeed impairs deep learning models. The
different roles of noise offer new explanations for deep models on specific
tasks and provide a new paradigm for improving model performance. Moreover, it
reminds us that we can influence the performance of learning systems via
information entropy change.

通过向输入 / 潜在特征添加噪声，探讨了熵变对深度学习系统的影响。实验结果表明，特定类型的噪声可以提升不同深度架构的性能，并通过降低信息熵来理论上证明增益，进而在大规模图像数据集上展示了显著的性能提升。