While pre-training large-scale video-language models (VLMs) has shown
remarkable potential for various downstream video-language tasks, existing VLMs
can still suffer from certain commonly seen limitations, e.g., coarse-grained
cross-modal aligning , under-modeling of temporal dynamics, detached
video-language view. In this work, we target enhancing VLMs with a fine-grained
structural spatio-temporal alignment learning method (namely Finsta). First of
all, we represent the input texts and videos with fine-grained scene graph (SG)
structures, both of which are further unified into a holistic SG (HSG) for
bridging two modalities. Then, an SG-based framework is built, where the
textual SG (TSG) is encoded with a graph Transformer, while the video dynamic
SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for
spatial and temporal feature propagation. A spatial-temporal Gaussian
differential graph Transformer is further devised to strengthen the sense of
the changes in objects across spatial and temporal dimensions. Next, based on
the fine-grained structural features of TSG and DSG, we perform object-centered
spatial alignment and predicate-centered temporal alignment respectively,
enhancing the video-language grounding in both the spatiality and temporality.
We design our method as a plug&play system, which can be integrated into
existing well-trained VLMs for further representation augmentation, without
training from scratch or relying on SG annotations in downstream applications.
On 6 representative VL modeling tasks over 12 datasets in both standard and
long-form video scenarios, Finsta consistently improves the existing 13
strong-performing VLMs persistently, and refreshes the current state-of-the-art
end task performance significantly in both the fine-tuning and zero-shot
settings.

通过精细化的结构化时空对齐学习方法（Finsta），将输入的文本和视频以细粒度场景图（SG）结构表示，进而统一为整体性 SG（HSG），从而加强语义和时序的视频 - 语言对齐，提高大规模视频 - 语言模型（VLMs）在各种下游任务中的性能。