We propose a Vision-Language Transformer (VLT) framework for referring
segmentation to facilitate deep interactions among multi-modal information and
enhance the holistic understanding to vision-language features. There are
different ways to understand the dynamic emphasis of a language expression,
especially when interacting with the image. However, the learned queries in
existing transformer works are fixed after training, which cannot cope with the
randomness and huge diversity of the language expressions. To address this
issue, we propose a Query Generation Module, which dynamically produces
multiple sets of input-specific queries to represent the diverse comprehensions
of language expression. To find the best among these diverse comprehensions, so
as to generate a better mask, we propose a Query Balance Module to selectively
fuse the corresponding responses of the set of queries. Furthermore, to enhance
the model's ability in dealing with diverse language expressions, we consider
inter-sample learning to explicitly endow the model with knowledge of
understanding different language expressions to the same object. We introduce
masked contrastive learning to narrow down the features of different
expressions for the same target object while distinguishing the features of
different objects. The proposed approach is lightweight and achieves new
state-of-the-art referring segmentation results consistently on five datasets.

该研究提出了一种轻量级的视觉 - 语言 Transformer 框架来实现引用分割，其中包括 Query Generation Module，Query Balance Module 以及 masked contrastive learning，它们能够动态生成多个特定于输入的查询，以明晰理解语言表达的多样性，同时针对不同的语言表达方式进行了明确的跨样本学习。该框架在五个数据集上都实现了最新的引用分割成果。

VLT: 面向参考分割的视觉语言变换器和查询生成

VLT: Vision-Language Transformer and Query Generation for Referring  Segmentation

A novel scene text recognizer based on Vision-Language Transformer (VLT) is
presented. Inspired by Levenshtein Transformer in the area of NLP, the proposed
method (named Levenshtein OCR, and LevOCR for short) explores an alternative
way for automatically transcribing textual content from cropped natural images.
Specifically, we cast the problem of scene text recognition as an iterative
sequence refinement process. The initial prediction sequence produced by a pure
vision model is encoded and fed into a cross-modal transformer to interact and
fuse with the visual features, to progressively approximate the ground truth.
The refinement process is accomplished via two basic character-level
operations: deletion and insertion, which are learned with imitation learning
and allow for parallel decoding, dynamic length change and good
interpretability. The quantitative experiments clearly demonstrate that LevOCR
achieves state-of-the-art performances on standard benchmarks and the
qualitative analyses verify the effectiveness and advantage of the proposed
LevOCR algorithm. Code is available at
this https URL

提出了一种基于 Vision-Language Transformer (VLT) 的新型场景文本识别器 (Levenshtein OCR)，利用交叉模态 Transformer 相互融合和逐步近似 ground truth, 通过模拟学习，实现并行解码，动态长度变化和良好的可解释性。实验证明，LevOCR 在标准基准测试中实现了最新水平，LevOCR 算法的有效性和优势得到了定量和定性分析的验证。