The computer vision community has seen a shift from convolutional-based to
pure transformer architectures for both image and video tasks. Training a
transformer from zero for these tasks usually requires a lot of data and
computational resources. Video Swin Transformer (VST) is a pure-transformer
model developed for video classification which achieves state-of-the-art
results in accuracy and efficiency on several datasets. In this paper, we aim
to understand if VST generalizes well enough to be used in an out-of-domain
setting. We study the performance of VST on two large-scale datasets, namely
FCVID and Something-Something using a transfer learning approach from
Kinetics-400, which requires around 4x less memory than training from scratch.
We then break down the results to understand where VST fails the most and in
which scenarios the transfer-learning approach is viable. Our experiments show
an 85\% top-1 accuracy on FCVID without retraining the whole model which is
equal to the state-of-the-art for the dataset and a 21\% accuracy on
Something-Something. The experiments also suggest that the performance of the
VST decreases on average when the video duration increases which seems to be a
consequence of a design choice of the model. From the results, we conclude that
VST generalizes well enough to classify out-of-domain videos without retraining
when the target classes are from the same type as the classes used to train the
model. We observed this effect when we performed transfer-learning from
Kinetics-400 to FCVID, where most datasets target mostly objects. On the other
hand, if the classes are not from the same type, then the accuracy after the
transfer-learning approach is expected to be poor. We observed this effect when
we performed transfer-learning from Kinetics-400, where the classes represent
mostly objects, to Something-Something, where the classes represent mostly
actions.

本文研究视频分类中使用 Video Swin Transformer 模型进行跨领域分类的泛化能力，使用 Kinetics-400 数据集进行迁移学习减少内存，实现 85% 的 top-1 准确率，但当对象不属于 Kinetics-400 数据集时准确率降低，为 21%。我们结论是，当目标类别与训练模型的类别相同时，VST 能够实现跨领域视频分类。