Orthogonality regularization has been developed to prevent deep CNNs from
training instability and feature redundancy. Among existing proposals, kernel
orthogonality regularization enforces orthogonality by minimizing the residual
between the Gram matrix formed by convolutional filters and the orthogonality
matrix.
We propose a novel measure for achieving better orthogonality among filters,
which disentangles diagonal and correlation information from the residual. The
model equipped with the measure under the principle of imposing strict
orthogonality between filters surpasses previous regularization methods in
near-orthogonality. Moreover, we observe the benefits of improved strict filter
orthogonality in relatively shallow models, but as model depth increases, the
performance gains in models employing strict kernel orthogonality decrease
sharply.
Furthermore, based on the observation of the potential conflict between
strict kernel orthogonality and growing model capacity, we propose a relaxation
theory on kernel orthogonality regularization. The relaxed kernel orthogonality
achieves enhanced performance on models with increased capacity, shedding light
on the burden of strict kernel orthogonality on deep model performance.
We conduct extensive experiments with our kernel orthogonality regularization
toolkit on ResNet and WideResNet in CIFAR-10 and CIFAR-100. We observe
state-of-the-art gains in model performance from the toolkit, which includes
both strict orthogonality and relaxed orthogonality regularization, and obtain
more robust models with expressive features. These experiments demonstrate the
efficacy of our toolkit and subtly provide insights into the often overlooked
challenges posed by strict orthogonality, addressing the burden of strict
orthogonality on capacity-rich models.

提出了一种用于改善卷积神经网络中的正交性的新方法，解决了传统正交性方法中对于模型容量提升的难题。通过我们的正交性偏差度量和我们的正交性放松理论，我们的工具包实现了最先进的性能和更加鲁棒的模型特征。