Vision-Language models have shown strong performance in the image-domain --
even in zero-shot settings, thanks to the availability of large amount of
pretraining data (i.e., paired image-text examples). However for videos, such
paired data is not as abundant. Thus, video-text models are usually designed by
adapting pretrained image-text models to video-domain, instead of training from
scratch. All such recipes rely on augmenting visual embeddings with temporal
information (i.e., image -> video), often keeping text embeddings unchanged or
even being discarded. In this paper, we argue that such adapted video-text
models can benefit more by augmenting text rather than visual information. We
propose VicTR, which jointly-optimizes text and video tokens, generating
'Video-conditioned Text' embeddings. Our method can further make use of
freely-available semantic information, in the form of visually-grounded
auxiliary text (e.g., object or scene information). We conduct experiments on
multiple benchmarks including supervised (Kinetics-400, Charades), zero-shot
and few-shot (HMDB-51, UCF-101) settings, showing competitive performance on
activity recognition based on video-text models.

本文提出了使用 VicTR 方法对视频文本模型进行优化，在视觉信息外，加入文本信息，以提高活动识别性能，实验结果证明在多个基准测试中，该方法具有竞争性能，特别是在视频文本模型的监督、零样本和少样本情况下。

VicTR: 视频条件的文本表示用于活动识别

VicTR: Video-conditioned Text Representations for Activity Recognition

Text-to-image multimodal tasks, generating/retrieving an image from a given
text description, are extremely challenging tasks since raw text descriptions
cover quite limited information in order to fully describe visually realistic
images. We propose a new visual contextual text representation for
text-to-image multimodal tasks, VICTR, which captures rich visual semantic
information of objects from the text input. First, we use the text description
as initial input and conduct dependency parsing to extract the syntactic
structure and analyse the semantic aspect, including object quantities, to
extract the scene graph. Then, we train the extracted objects, attributes, and
relations in the scene graph and the corresponding geometric relation
information using Graph Convolutional Networks, and it generates text
representation which integrates textual and visual semantic information. The
text representation is aggregated with word-level and sentence-level embedding
to generate both visual contextual word and sentence representation. For the
evaluation, we attached VICTR to the state-of-the-art models in text-to-image
generation.VICTR is easily added to existing models and improves across both
quantitative and qualitative aspects.

本文提出了一种新的视觉上下文文本表示方法，VICTR，用于处理文本到图像的多模态任务，通过使用图卷积网络和文本表征的结合，有效地捕捉了文本语义中的视觉特征信息，实现了在实验中得到的量化和定性的改进。