Sign language, which conveys meaning through gestures, is the chief means of
communication among deaf people. Recognizing sign language in natural settings
presents significant challenges due to factors such as lighting, background
clutter, and variations in signer characteristics. In this thesis, I study
automatic sign language processing in the wild, using signing videos collected
from the Internet. This thesis contributes new datasets, tasks, and methods.
Most chapters of this thesis address tasks related to fingerspelling, an
important component of sign language and yet has not been studied widely by
prior work. I present three new large-scale ASL datasets in the wild:
ChicagoFSWild, ChicagoFSWild+, and OpenASL. Using ChicagoFSWild and
ChicagoFSWild+, I address fingerspelling recognition, which consists of
transcribing fingerspelling sequences into text. I propose an end-to-end
approach based on iterative attention that allows recognition from a raw video
without explicit hand detection. I further show that using a Conformer-based
network jointly modeling handshape and mouthing can bring performance close to
that of humans. Next, I propose two tasks for building real-world
fingerspelling-based applications: fingerspelling detection and search. For
fingerspelling detection, I introduce a suite of evaluation metrics and a new
detection model via multi-task training. To address the problem of searching
for fingerspelled keywords in raw sign language videos, we propose a novel
method that jointly localizes and matches fingerspelling segments to text.
Finally, I will describe a benchmark for large-vocabulary open-domain sign
language translation based on OpenASL. To address the challenges of sign
language translation in realistic settings, we propose a set of techniques
including sign search as a pretext task for pre-training and fusion of mouthing
and handshape features.

这篇论文研究了自然环境下手语的自动处理，包括手指拼写、手势识别和手语翻译，并提出了新的数据集、任务和方法。

走向现实世界中的美国手语处理：数据、任务和方法

Toward American Sign Language Processing in the Real World: Data, Tasks,  and Methods

Fingerspelling, in which words are signed letter by letter, is an important
component of American Sign Language. Most previous work on automatic
fingerspelling recognition has assumed that the boundaries of fingerspelling
regions in signing videos are known beforehand. In this paper, we consider the
task of fingerspelling detection in raw, untrimmed sign language videos. This
is an important step towards building real-world fingerspelling recognition
systems. We propose a benchmark and a suite of evaluation metrics, some of
which reflect the effect of detection on the downstream fingerspelling
recognition task. In addition, we propose a new model that learns to detect
fingerspelling via multi-task training, incorporating pose estimation and
fingerspelling recognition (transcription) along with detection, and compare
this model to several alternatives. The model outperforms all alternative
approaches across all metrics, establishing a state of the art on the
benchmark.

本文提出在原始的、未修剪的手语视频中检测指拼的新模型，采用多任务训练方法，同时融合姿态估计和指拼转录，相比其他方法表现得更优，为建立实际应用的指拼识别系统提供了重要的步骤和技术。

美国手语拼写检测

Fingerspelling Detection in American Sign Language

Sign language recognition is a challenging gesture sequence recognition
problem, characterized by quick and highly coarticulated motion. In this paper
we focus on recognition of fingerspelling sequences in American Sign Language
(ASL) videos collected in the wild, mainly from YouTube and Deaf social media.
Most previous work on sign language recognition has focused on controlled
settings where the data is recorded in a studio environment and the number of
signers is limited. Our work aims to address the challenges of real-life data,
reducing the need for detection or segmentation modules commonly used in this
domain. We propose an end-to-end model based on an iterative attention
mechanism, without explicit hand detection or segmentation. Our approach
dynamically focuses on increasingly high-resolution regions of interest. It
outperforms prior work by a large margin. We also introduce a newly collected
data set of crowdsourced annotations of fingerspelling in the wild, and show
that performance can be further improved with this additional data set.

这篇研究提出了一种基于递归注意机制的端到端模型，用于在野外 ASL 视频中识别手语拼写序列。新的数据集的引入提高了性能。

野外手语识别及迭代视觉注意力

Fingerspelling recognition in the wild with iterative visual attention

We study the problem of recognizing video sequences of fingerspelled letters
in American Sign Language (ASL). Fingerspelling comprises a significant but
relatively understudied part of ASL. Recognizing fingerspelling is challenging
for a number of reasons: It involves quick, small motions that are often highly
coarticulated; it exhibits significant variation between signers; and there has
been a dearth of continuous fingerspelling data collected. In this work we
collect and annotate a new data set of continuous fingerspelling videos,
compare several types of recognizers, and explore the problem of signer
variation. Our best-performing models are segmental (semi-Markov) conditional
random fields using deep neural network-based features. In the signer-dependent
setting, our recognizers achieve up to about 92% letter accuracy. The
multi-signer setting is much more challenging, but with neural network
adaptation we achieve up to 83% letter accuracies in this setting.

研究美国手语中指拼字字母的视频序列识别问题，并通过收集和注释新的连续指拼视频数据集，比较多种类型的识别器，探索签名者差异问题。基于深度神经网络的特征，我们的半马尔可夫条件随机场模型在签名者依赖和多签名者情景下均取得较高的字母识别准确率。