Spatial convolutions are extensively used in numerous deep video models. It
fundamentally assumes spatio-temporal invariance, i.e., using shared weights
for every location in different frames. This work presents Temporally-Adaptive
Convolutions (TAdaConv) for video understanding, which shows that adaptive
weight calibration along the temporal dimension is an efficient way to
facilitate modeling complex temporal dynamics in videos. Specifically, TAdaConv
empowers spatial convolutions with temporal modeling abilities by calibrating
the convolution weights for each frame according to its local and global
temporal context. Compared to existing operations for temporal modeling,
TAdaConv is more efficient as it operates over the convolution kernels instead
of the features, whose dimension is an order of magnitude smaller than the
spatial resolutions. Further, kernel calibration brings an increased model
capacity. Based on this readily plug-in operation TAdaConv as well as its
extension, i.e., TAdaConvV2, we construct TAdaBlocks to empower ConvNeXt and
Vision Transformer to have strong temporal modeling capabilities. Empirical
results show TAdaConvNeXtV2 and TAdaFormer perform competitively against
state-of-the-art convolutional and Transformer-based models in various video
understanding benchmarks. Our codes and models are released at:
this https URL

通过根据局部和全局时间上下文校准卷积权重，将时空卷积赋予时间建模能力，从而为视频理解提供效率更高、模型容量更大的 TAdaConv，并通过 TAdaConvNeXtV2 和 TAdaFormer 在各种视频理解评估中取得与最先进的基于卷积和 Transformer 的模型相竞争的实证结果。

高效视频理解的时间自适应模型

Temporally-Adaptive Models for Efficient Video Understanding

Spatial convolutions are widely used in numerous deep video models. It
fundamentally assumes spatio-temporal invariance, i.e., using shared weights
for every location in different frames. This work presents Temporally-Adaptive
Convolutions (TAdaConv) for video understanding, which shows that adaptive
weight calibration along the temporal dimension is an efficient way to
facilitate modelling complex temporal dynamics in videos. Specifically,
TAdaConv empowers the spatial convolutions with temporal modelling abilities by
calibrating the convolution weights for each frame according to its local and
global temporal context. Compared to previous temporal modelling operations,
TAdaConv is more efficient as it operates over the convolution kernels instead
of the features, whose dimension is an order of magnitude smaller than the
spatial resolutions. Further, the kernel calibration brings an increased model
capacity. We construct TAda2D and TAdaConvNeXt networks by replacing the 2D
convolutions in ResNet and ConvNeXt with TAdaConv, which leads to at least on
par or better performance compared to state-of-the-art approaches on multiple
video action recognition and localization benchmarks. We also demonstrate that
as a readily plug-in operation with negligible computation overhead, TAdaConv
can effectively improve many existing video models with a convincing margin.

本研究提出一种名为时域自适应卷积 (TAdaConv) 的视频理解技术，通过根据局部和全局时间上下文为每个帧调整卷积权重，使空间卷积具备时间建模能力，从而实现了对视频中复杂时序动态的有效建模和优化，并在多个视频动作识别和定位基准上实现了与现有先进方法至少一致或更好的表现。

TAda！基于时间自适应卷积的视频理解

TAda! Temporally-Adaptive Convolutions for Video Understanding

The shift operation was recently introduced as an alternative to spatial
convolutions. The operation moves subsets of activations horizontally and/or
vertically. Spatial convolutions are then replaced with shift operations
followed by point-wise convolutions, significantly reducing computational
costs. In this work, we investigate how shifts should best be applied to high
accuracy CNNs. We apply shifts of two different neighbourhood groups to ResNet
on ImageNet: the originally introduced 8-connected (8C) neighbourhood shift and
the less well studied 4-connected (4C) neighbourhood shift. We find that when
replacing ResNet's spatial convolutions with shifts, both shift neighbourhoods
give equal ImageNet accuracy, showing the sufficiency of small neighbourhoods
for large images. Interestingly, when incorporating shifts to all point-wise
convolutions in residual networks, 4-connected shifts outperform 8-connected
shifts. Such a 4-connected shift setup gives the same accuracy as full residual
networks while reducing the number of parameters and FLOPs by over 40%. We then
highlight that without spatial convolutions, ResNet's downsampling/upsampling
bottleneck channel structure is no longer needed. We show a new, 4C shift-based
residual network, much shorter than the original ResNet yet with a higher
accuracy for the same computational cost. This network is the highest accuracy
shift-based network yet shown, demonstrating the potential of shifting in deep
neural networks.

本文研究了如何应用转移运算以替代空间卷积操作，通过将 8-connected 和 4-connected 邻域组的转移操作应用于 ResNet 的 point-wise 卷积中，实现了与全残差网络相同的准确性，同时将参数和计算要求降低了 40％，通过减少横向连接层数来生成了一个准确性更高的新 ResNet 网络。

4 连通移位残差网络

4-Connected Shift Residual Networks

Neural networks rely on convolutions to aggregate spatial information.
However, spatial convolutions are expensive in terms of model size and
computation, both of which grow quadratically with respect to kernel size. In
this paper, we present a parameter-free, FLOP-free "shift" operation as an
alternative to spatial convolutions. We fuse shifts and point-wise convolutions
to construct end-to-end trainable shift-based modules, with a hyperparameter
characterizing the tradeoff between accuracy and efficiency. To demonstrate the
operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based
modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters;
we additionally demonstrate the operation's resilience to parameter reduction
on ImageNet, outperforming ResNet family members. We finally show the shift
operation's applicability across domains, achieving strong performance with
fewer parameters on classification, face verification and style transfer.

本文提出一种基于 shift 的操作，用于代替 spatial convolutions 提高神经网络的效率，并在分类、人脸识别、风格迁移等各类领域中取得了不错的表现。