The canonical approach to video captioning dictates a caption generation
model to learn from offline-extracted dense video features. These feature
extractors usually operate on video frames sampled at a fixed frame rate and
are often trained on image/video understanding tasks, without adaption to video
captioning data. In this work, we present SwinBERT, an end-to-end
transformer-based model for video captioning, which takes video frame patches
directly as inputs, and outputs a natural language description. Instead of
leveraging multiple 2D/3D feature extractors, our method adopts a video
transformer to encode spatial-temporal representations that can adapt to
variable lengths of video input without dedicated design for different frame
rates. Based on this model architecture, we show that video captioning can
benefit significantly from more densely sampled video frames as opposed to
previous successes with sparsely sampled video frames for video-and-language
understanding tasks (e.g., video question answering). Moreover, to avoid the
inherent redundancy in consecutive video frames, we propose adaptively learning
a sparse attention mask and optimizing it for task-specific performance
improvement through better long-range video sequence modeling. Through
extensive experiments on 5 video captioning datasets, we show that SwinBERT
achieves across-the-board performance improvements over previous methods, often
by a large margin. The learned sparse attention masks in addition push the
limit to new state of the arts, and can be transferred between different video
lengths and between different datasets. Code is available at
this https URL

本文提出了一种基于 SwineBERT 的视频字幕生成模型，该模型通过对稠密采样的视频帧进行变换来进行自适应学习，同时通过自适应学习稀疏注意力掩码来对长序列视频进行建模以实现任务性能提升，在五个电影字幕数据集上取得了显著的性能改进和新的最佳结果。

SwinBERT：基于稀疏注意力的端到端变压器模型用于视频字幕生成

SwinBERT: End-to-End Transformers with Sparse Attention for Video  Captioning

Our objective in this work is fine-grained classification of actions in
untrimmed videos, where the actions may be temporally extended or may span only
a few frames of the video. We cast this into a query-response mechanism, where
each query addresses a particular question, and has its own response label set.
We make the following four contributions: (I) We propose a new model - a
Temporal Query Network - which enables the query-response functionality, and a
structural understanding of fine-grained actions. It attends to relevant
segments for each query with a temporal attention mechanism, and can be trained
using only the labels for each query. (ii) We propose a new way - stochastic
feature bank update - to train a network on videos of various lengths with the
dense sampling required to respond to fine-grained queries. (iii) We compare
the TQN to other architectures and text supervision methods, and analyze their
pros and cons. Finally, (iv) we evaluate the method extensively on the FineGym
and Diving48 benchmarks for fine-grained action classification and surpass the
state-of-the-art using only RGB features.

本文介绍了一种基于 Temporal Query Network 的时空查询机制，通过稠密采样与随机特征库更新方式实现了对细粒度行为的分类，最终在 FineGym 和 Diving48 的数据集上达到了当前最佳效果。

时间查询网络用于细粒度视频理解

Temporal Query Networks for Fine-grained Video Understanding

We introduce and tackle the problem of zero-shot object detection (ZSD),
which aims to detect object classes which are not observed during training. We
work with a challenging set of object classes, not restricting ourselves to
similar and/or fine-grained categories as in prior works on zero-shot
classification. We present a principled approach by first adapting
visual-semantic embeddings for ZSD. We then discuss the problems associated
with selecting a background class and motivate two background-aware approaches
for learning robust detectors. One of these models uses a fixed background
class and the other is based on iterative latent assignments. We also outline
the challenge associated with using a limited number of training classes and
propose a solution based on dense sampling of the semantic label space using
auxiliary data with a large number of categories. We propose novel splits of
two standard detection datasets - MSCOCO and VisualGenome, and present
extensive empirical results in both the traditional and generalized zero-shot
settings to highlight the benefits of the proposed methods. We provide useful
insights into the algorithm and conclude by posing some open questions to
encourage further research.

本文提出了一种解决零样本目标检测问题的方法，并通过使用视觉 - 语义嵌入，背景感知方法，以及使用大量类别语义标签的辅助数据来解决仅有少数训练类别的问题。在 MSCOCO 和 VisualGenome 两个标准检测数据集上进行实验，验证了该方法的有效性。