This paper describes our champion solution for the CVPR2022 Generic Event
Boundary Captioning (GEBC) competition. GEBC requires the captioning model to
have a comprehension of instantaneous status changes around the given video
boundary, which makes it much more challenging than conventional video
captioning task. In this paper, a Dual-Stream Transformer with improvements on
both video content encoding and captions generation is proposed: (1) We utilize
three pre-trained models to extract the video features from different
granularities. Moreover, we exploit the types of boundary as hints to help the
model generate captions. (2) We particularly design an model, termed as
Dual-Stream Transformer, to learn discriminative representations for boundary
captioning. (3) Towards generating content-relevant and human-like captions, we
improve the description quality by designing a word-level ensemble strategy.
The promising results on the GEBC test split demonstrate the efficacy of our
proposed model.

本文介绍了我们在 CVPR2022 Generic Event Boundary Captioning 竞赛中的冠军方案，提出了一种名为 Dual-Stream Transformer 的模型，利用三个预训练模型从不同颗粒度提取视频特征以辅助生成说说内容，设计了词级集成策略以提升生成质量，在 GEBC 测试集上取得了良好的效果。

用于通用事件边界字幕生成的双流 Transformer

Dual-Stream Transformer for Generic Event Boundary Captioning

Modern high-scoring models of vision in the brain score competition do not
stem from Vision Transformers. However, in this paper, we provide evidence
against the unexpected trend of Vision Transformers (ViT) being not
perceptually aligned with human visual representations by showing how a
dual-stream Transformer, a CrossViT$~\textit{a la}$ Chen et al. (2021), under a
joint rotationally-invariant and adversarial optimization procedure yields 2nd
place in the aggregate Brain-Score 2022 competition(Schrimpf et al., 2020b)
averaged across all visual categories, and at the time of the competition held
1st place for the highest explainable variance of area V4. In addition, our
current Transformer-based model also achieves greater explainable variance for
areas V4, IT and Behaviour than a biologically-inspired CNN (ResNet50) that
integrates a frontal V1-like computation module (Dapello et al.,2020). To
assess the contribution of the optimization scheme with respect to the CrossViT
architecture, we perform several additional experiments on differently
optimized CrossViT's regarding adversarial robustness, common corruption
benchmarks, mid-ventral stimuli interpretation and feature inversion. Against
our initial expectations, our family of results provides tentative support for
an $\textit{"All roads lead to Rome"}$ argument enforced via a joint
optimization rule even for non biologically-motivated models of vision such as
Vision Transformers. Code is available at
this https URL

该论文通过展示如何使用双流 Transformer 和联合旋转不变性和对抗优化过程，实现了在 Brain-Score 2022 竞赛中排名第二，并在所有视觉类别中平均排名第一的第一个位置，并且在 V4 区域的可解释方差方面取得了最高的成绩。同时，该模型在 V4，IT 和 Behavior 方面的解释方差也优于 ResNet50 模型。此外，通过几项额外实验，论文支持了一个 “所有道路通往罗马” 的论点，这是通过联合优化规则来实现的，即使对于像 Vision Transformers 这样的非生物学模型。