Transformer achieves remarkable successes in understanding 1 and
2-dimensional signals (e.g., NLP and Image Content Understanding). As a
potential alternative to convolutional neural networks, it shares merits of
strong interpretability, high discriminative power on hyper-scale data, and
flexibility in processing varying length inputs. However, its encoders
naturally contain computational intensive operations such as pair-wise
self-attention, incurring heavy computational burden when being applied on the
complex 3-dimensional video signals.
This paper presents Token Shift Module (i.e., TokShift), a novel,
zero-parameter, zero-FLOPs operator, for modeling temporal relations within
each transformer encoder. Specifically, the TokShift barely temporally shifts
partial [Class] token features back-and-forth across adjacent frames. Then, we
densely plug the module into each encoder of a plain 2D vision transformer for
learning 3D video representation. It is worth noticing that our TokShift
transformer is a pure convolutional-free video transformer pilot with
computational efficiency for video understanding. Experiments on standard
benchmarks verify its robustness, effectiveness, and efficiency. Particularly,
with input clips of 8/12 frames, the TokShift transformer achieves SOTA
precision: 79.83%/80.40% on the Kinetics-400, 66.56% on EGTEA-Gaze+, and 96.80%
on UCF-101 datasets, comparable or better than existing SOTA convolutional
counterparts. Our code is open-sourced in:
this https URL

提出了一种基于 Token Shift Module 的 Transformer 操作，用于学习 3D 视频表示，以在消除卷积神经网络的同时具有计算效率。实验证明，该方法比现有的卷积方法取得更好的结果。

视频分类的 Token Shift Transformer

Token Shift Transformer for Video Classification

The progress in autonomous driving is also due to the increased availability
of vast amounts of training data for the underlying machine learning
approaches. Machine learning systems are generally known to lack robustness,
e.g., if the training data did rarely or not at all cover critical situations.
The challenging task of corner case detection in video, which is also somehow
related to unusual event or anomaly detection, aims at detecting these unusual
situations, which could become critical, and to communicate this to the
autonomous driving system (online use case). Such a system, however, could be
also used in offline mode to screen vast amounts of data and select only the
relevant situations for storing and (re)training machine learning algorithms.
So far, the approaches for corner case detection have been limited to videos
recorded from a fixed camera, mostly for security surveillance. In this paper,
we provide a formal definition of a corner case and propose a system framework
for both the online and the offline use case that can handle video signals from
front cameras of a naturally moving vehicle and can output a corner case score.

该研究论文提供了 “corner case” 的定义，并提出了一种既可在线使用，也可脱机筛选、重复训练机器学习算法的系统框架，该系统能够处理自然移动车辆前置摄像头的视频信号，并输出 “corner case” 分数。