Sign language, which conveys meaning through gestures, is the chief means of
communication among deaf people. Recognizing sign language in natural settings
presents significant challenges due to factors such as lighting, background
clutter, and variations in signer characteristics. In this thesis, I study
automatic sign language processing in the wild, using signing videos collected
from the Internet. This thesis contributes new datasets, tasks, and methods.
Most chapters of this thesis address tasks related to fingerspelling, an
important component of sign language and yet has not been studied widely by
prior work. I present three new large-scale ASL datasets in the wild:
ChicagoFSWild, ChicagoFSWild+, and OpenASL. Using ChicagoFSWild and
ChicagoFSWild+, I address fingerspelling recognition, which consists of
transcribing fingerspelling sequences into text. I propose an end-to-end
approach based on iterative attention that allows recognition from a raw video
without explicit hand detection. I further show that using a Conformer-based
network jointly modeling handshape and mouthing can bring performance close to
that of humans. Next, I propose two tasks for building real-world
fingerspelling-based applications: fingerspelling detection and search. For
fingerspelling detection, I introduce a suite of evaluation metrics and a new
detection model via multi-task training. To address the problem of searching
for fingerspelled keywords in raw sign language videos, we propose a novel
method that jointly localizes and matches fingerspelling segments to text.
Finally, I will describe a benchmark for large-vocabulary open-domain sign
language translation based on OpenASL. To address the challenges of sign
language translation in realistic settings, we propose a set of techniques
including sign search as a pretext task for pre-training and fusion of mouthing
and handshape features.

这篇论文研究了自然环境下手语的自动处理，包括手指拼写、手势识别和手语翻译，并提出了新的数据集、任务和方法。