This work is on training a generative action/video recognition model whose
output is a free-form action-specific caption describing the video (rather than
an action class label). A generative approach has practical advantages like
producing more fine-grained and human-readable output, and being naturally
open-world. To this end, we propose to adapt a pre-trained generative Vision &
Language (V&L) Foundation Model for video/action recognition. While recently
there have been a few attempts to adapt V&L models trained with contrastive
learning (e.g. CLIP) for video/action, to the best of our knowledge, we propose
the very first method that sets outs to accomplish this goal for a generative
model. We firstly show that direct fine-tuning of a generative model to produce
action classes suffers from severe overfitting. To alleviate this, we introduce
REST, a training framework consisting of two key components: an unsupervised
method for adapting the generative model to action/video by means of
pseudo-caption generation and Self-training, i.e. without using any
action-specific labels; (b) a Retrieval approach based on CLIP for discovering
a diverse set of pseudo-captions for each video to train the model.
Importantly, we show that both components are necessary to obtain high
accuracy. We evaluate REST on the problem of zero-shot action recognition where
we show that our approach is very competitive when compared to contrastive
learning-based methods. Code will be made available.

本文提出使用 REST 框架，在不使用任何动作特定标签的情况下，通过生成的方式将生成模型适应于影片 / 动作，并使用基于 CLIP 的检索方法来训练模型，旨在实现训练一个生成模型，生成一个动画特定的自由格式标题来描述图像 / 动画识别，以解决细粒度识别难以在传统的目录式方法中表示的问题，并取得了不错的成效。