In this paper, we tackle the problem of sign language translation (SLT)
without gloss annotations. Although intermediate representation like gloss has
been proven effective, gloss annotations are hard to acquire, especially in
large quantities. This limits the domain coverage of translation datasets, thus
handicapping real-world applications. To mitigate this problem, we design the
Gloss-Free End-to-end sign language translation framework (GloFE). Our method
improves the performance of SLT in the gloss-free setting by exploiting the
shared underlying semantics of signs and the corresponding spoken translation.
Common concepts are extracted from the text and used as a weak form of
intermediate representation. The global embedding of these concepts is used as
a query for cross-attention to find the corresponding information within the
learned visual features. In a contrastive manner, we encourage the similarity
of query results between samples containing such concepts and decrease those
that do not. We obtained state-of-the-art results on large-scale datasets,
including OpenASL and How2Sign. The code and model will be available at
this https URL

本文介绍了 GloFE 框架，该框架解决了无手语注释的手语翻译问题，并通过利用手语和口语的共同语义以及密码比对，在包括 OpenASL 和 How2Sign 在内的大型数据集上获得了最新的结果。

非干涉式端到端手语翻译

Gloss-Free End-to-End Sign Language Translation

We propose Dual Attention Networks (DANs) which jointly leverage visual and
textual attention mechanisms to capture fine-grained interplay between vision
and language. DANs attend to specific regions in images and words in text
through multiple steps and gather essential information from both modalities.
Based on this framework, we introduce two types of DANs for multimodal
reasoning and matching, respectively. The reasoning model allows visual and
textual attentions to steer each other during collaborative inference, which is
useful for tasks such as Visual Question Answering (VQA). In addition, the
matching model exploits the two attention mechanisms to estimate the similarity
between images and sentences by focusing on their shared semantics. Our
extensive experiments validate the effectiveness of DANs in combining vision
and language, achieving the state-of-the-art performance on public benchmarks
for VQA and image-text matching.

提出双重关注网络框架（Dual Attention Networks），应用于视觉和文本场景的多模态推理、比对任务，在 Visual Question Answering 和图片 - 文本比对任务中取得最新的优异表现。