Human body trajectories are a salient cue to identify actions in the video.
Such body trajectories are mainly conveyed by hands and face across consecutive
frames in sign language. However, current methods in continuous sign language
recognition (CSLR) usually process frames independently, thus failing to
capture cross-frame trajectories to effectively identify a sign. To handle this
limitation, we propose correlation network (CorrNet) to explicitly capture and
leverage body trajectories across frames to identify signs. In specific, a
correlation module is first proposed to dynamically compute correlation maps
between the current frame and adjacent frames to identify trajectories of all
spatial patches. An identification module is then presented to dynamically
emphasize the body trajectories within these correlation maps. As a result, the
generated features are able to gain an overview of local temporal movements to
identify a sign. Thanks to its special attention on body trajectories, CorrNet
achieves new state-of-the-art accuracy on four large-scale datasets, i.e.,
PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. A comprehensive comparison with
previous spatial-temporal reasoning methods verifies the effectiveness of
CorrNet. Visualizations demonstrate the effects of CorrNet on emphasizing human
body trajectories across adjacent frames.

提出一种卷积神经网络 CorrNet 来识别手语，并结合当前帧和相邻帧之间的人体轨迹，从而捕捉视频中人体动作的局部时空特征，进而在大型数据集中获得最先进的准确率。