In sign language, the conveyance of human body trajectories predominantly
relies upon the coordinated movements of hands and facial expressions across
successive frames. Despite the recent advancements of sign language
understanding methods, they often solely focus on individual frames, inevitably
overlooking the inter-frame correlations that are essential for effectively
modeling human body trajectories. To address this limitation, this paper
introduces a spatial-temporal correlation network, denoted as CorrNet+, which
explicitly identifies body trajectories across multiple frames. In specific,
CorrNet+ employs a correlation module and an identification module to build
human body trajectories. Afterwards, a temporal attention module is followed to
adaptively evaluate the contributions of different frames. The resultant
features offer a holistic perspective on human body movements, facilitating a
deeper understanding of sign language. As a unified model, CorrNet+ achieves
new state-of-the-art performance on two extensive sign language understanding
tasks, including continuous sign language recognition (CSLR) and sign language
translation (SLT). Especially, CorrNet+ surpasses previous methods equipped
with resource-intensive pose-estimation networks or pre-extracted heatmaps for
hand and facial feature extraction. Compared with CorrNet, CorrNet+ achieves a
significant performance boost across all benchmarks while halving the
computational overhead. A comprehensive comparison with previous
spatial-temporal reasoning methods verifies the superiority of CorrNet+. Code
is available at this https URL

本文介绍了一种称为 CorrNet + 的空间 - 时间相关网络，它明确识别多帧中的人体轨迹，为进行手语的深入理解提供全面的视角。作为一个统一模型，CorrNet + 在连续手语识别（CSLR）和手语翻译（SLT）两个广泛的手语理解任务中实现了新的最先进性能，同时减少了一半的计算开销。

CorrNet+: 空间 - 时间相关性的手语识别和翻译

CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal  Correlation

Human body trajectories are a salient cue to identify actions in the video.
Such body trajectories are mainly conveyed by hands and face across consecutive
frames in sign language. However, current methods in continuous sign language
recognition (CSLR) usually process frames independently, thus failing to
capture cross-frame trajectories to effectively identify a sign. To handle this
limitation, we propose correlation network (CorrNet) to explicitly capture and
leverage body trajectories across frames to identify signs. In specific, a
correlation module is first proposed to dynamically compute correlation maps
between the current frame and adjacent frames to identify trajectories of all
spatial patches. An identification module is then presented to dynamically
emphasize the body trajectories within these correlation maps. As a result, the
generated features are able to gain an overview of local temporal movements to
identify a sign. Thanks to its special attention on body trajectories, CorrNet
achieves new state-of-the-art accuracy on four large-scale datasets, i.e.,
PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. A comprehensive comparison with
previous spatial-temporal reasoning methods verifies the effectiveness of
CorrNet. Visualizations demonstrate the effects of CorrNet on emphasizing human
body trajectories across adjacent frames.

提出一种卷积神经网络 CorrNet 来识别手语，并结合当前帧和相邻帧之间的人体轨迹，从而捕捉视频中人体动作的局部时空特征，进而在大型数据集中获得最先进的准确率。