Downsampling operators break the shift invariance of convolutional neural
networks (CNNs) and this affects the robustness of features learned by CNNs
when dealing with even small pixel-level shift. Through a large-scale
correlation analysis framework, we study shift invariance of CNNs by inspecting
existing downsampling operators in terms of their maximum-sampling bias (MSB),
and find that MSB is negatively correlated with shift invariance. Based on this
crucial insight, we propose a learnable pooling operator called Translation
Invariant Polyphase Sampling (TIPS) and two regularizations on the intermediate
feature maps of TIPS to reduce MSB and learn translation-invariant
representations. TIPS can be integrated into any CNN and can be trained
end-to-end with marginal computational overhead. Our experiments demonstrate
that TIPS results in consistent performance gains in terms of accuracy, shift
consistency, and shift fidelity on multiple benchmarks for image classification
and semantic segmentation compared to previous methods and also leads to
improvements in adversarial and distributional robustness. TIPS results in the
lowest MSB compared to all previous methods, thus explaining our strong
empirical results.

基于大规模相关性分析框架，通过研究现有的下采样算子，我们发现最大采样偏差与转移不变性呈负相关。基于这一关键洞察，我们提出了一种可学习的池化算子，称为平移不变的多相采样 (TIPS)，并对 TIPS 的中间特征映射进行了两种正则化以减小最大采样偏差，从而学习到转移不变的表示。我们的实验证明，与之前的方法相比，TIPS 在多个图像分类和语义分割基准测试中都取得了持续的性能提升，并且在对抗性和分布鲁棒性方面也有所改进。TIPS 相较于之前的方法具有最低的最大采样偏差，从而解释了我们的强实证结果。