Convolutional Neural Networks (CNNs) have advanced significantly in visual
representation learning and recognition. However, they face notable challenges
in performance and computational efficiency when dealing with real-world,
multi-scale image inputs. Conventional methods rescale all input images into a
fixed size, wherein a larger fixed size favors performance but rescaling small
size images to a larger size incurs digitization noise and increased
computation cost. In this work, we carry out a comprehensive, layer-wise
investigation of CNN models in response to scale variation, based on Centered
Kernel Alignment (CKA) analysis. The observations reveal lower layers are more
sensitive to input image scale variations than high-level layers. Inspired by
this insight, we propose Multi-scale Unified Network (MUSN) consisting of
multi-scale subnets, a unified network, and scale-invariant constraint. Our
method divides the shallow layers into multi-scale subnets to enable feature
extraction from multi-scale inputs, and the low-level features are unified in
deep layers for extracting high-level semantic features. A scale-invariant
constraint is posed to maintain feature consistency across different scales.
Extensive experiments on ImageNet and other scale-diverse datasets, demonstrate
that MSUN achieves significant improvements in both model performance and
computational efficiency. Particularly, MSUN yields an accuracy increase up to
44.53% and diminishes FLOPs by 7.01-16.13% in multi-scale scenarios.

在这项工作中，我们基于中心核对齐（CKA）分析对卷积神经网络（CNNs）的模型进行了全面的分层调查，结果表明较低的层对输入图像尺度变化更敏感。在此基础上，我们提出了多尺度统一网络（MUSN），由多尺度子网络、一个统一网络和尺度不变约束组成，以获得从多尺度输入中提取特征的能力，并将低级特征在深层进行统一，提取高层语义特征。大量在 ImageNet 和其他尺度多样的数据集上进行的实验表明，MUSN 在模型性能和计算效率方面取得了显著的改进。特别是在多尺度情景下，MUSN 的准确率提高了 44.53％，FLOPs 降低了 7.01-16.13％。