Cross-modal alignment plays a crucial role in vision-language pre-training
(VLP) models, enabling them to capture meaningful associations across different
modalities. For this purpose, inspired by the success of masked language
modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling
tasks have been proposed for VLP to further promote cross-modal interactions.
The core idea of previous masked modeling tasks is to focus on reconstructing
the masked tokens based on visible context for learning local-local alignment,
i.e., associations between image patches and text tokens. However, most of them
pay little attention to the global semantic features generated for the masked
data, resulting in a limited cross-modal alignment ability of global
representations to local features of the other modality. Therefore, in this
paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL)
task to facilitate global-local alignment and local-local alignment
simultaneously. Specifically, the GLSCL task complements the missing semantics
of masked data and recovers global and local features by cross-modal
interactions. Our GLSCL consists of masked global semantic completion (MGSC)
and masked local token completion (MLTC). MGSC promotes learning more
representative global features which have a great impact on the performance of
downstream tasks, and MLTC can further enhance accurate comprehension on
multimodal data. Moreover, we present a flexible vision encoder, enabling our
model to simultaneously perform image-text and video-text multimodal tasks.
Experimental results show that our proposed method obtains state-of-the-art
performance on various vision-language benchmarks, such as visual question
answering, image-text retrieval, and video-text retrieval.

本文提出了一种 GLSCL 任务，旨在促进全局 - 局部对齐和局部 - 局部对齐，该任务包括 MGSC 和 MLTC，可通过跨模式交互补充掩码数据的缺失语义并恢复全局和局部特征，实验结果显示，该方法在多种视觉语言基准测试中获得了最先进的性能。

视觉语言预训练的全局和局部语义补全学习

Global and Local Semantic Completion Learning for Vision-Language  Pre-training

In this paper, we propose a novel learning-based polygonal point set tracking
method. Compared to existing video object segmentation~(VOS) methods that
propagate pixel-wise object mask information, we propagate a polygonal point
set over frames.
Specifically, the set is defined as a subset of points in the target contour,
and our goal is to track corresponding points on the target contour. Those
outputs enable us to apply various visual effects such as motion tracking, part
deformation, and texture mapping. To this end, we propose a new method to track
the corresponding points between frames by the global-local alignment with
delicately designed losses and regularization terms. We also introduce a novel
learning strategy using synthetic and VOS datasets that makes it possible to
tackle the problem without developing the point correspondence dataset. Since
the existing datasets are not suitable to validate our method, we build a new
polygonal point set tracking dataset and demonstrate the superior performance
of our method over the baselines and existing contour-based VOS methods. In
addition, we present visual-effects applications of our method on part
distortion and text mapping.

本文提出了一种基于学习的多边形点集跟踪方法，通过全局 - 局部对齐和精细设计的损失和正则化项，以多边形点集方式传播目标轮廓，从而实现运动跟踪、部分变形和纹理映射等多种视觉效果。同时通过 VOS 数据集的学习策略，避免了点对应数据集的开发，并在新的跟踪数据集上证明了该方法的优异性能。