Continuous sign language recognition (SLR) deals with unaligned video-text
pair and uses the word error rate (WER), i.e., edit distance, as the main
evaluation metric. Since it is not differentiable, we usually instead optimize
the learning model with the connectionist temporal classification (CTC)
objective loss, which maximizes the posterior probability over the sequential
alignment. Due to the optimization gap, the predicted sentence with the highest
decoding probability may not be the best choice under the WER metric. To tackle
this issue, we propose a novel architecture with cross modality augmentation.
Specifically, we first augment cross-modal data by simulating the calculation
procedure of WER, i.e., substitution, deletion and insertion on both text label
and its corresponding video. With these real and generated pseudo video-text
pairs, we propose multiple loss terms to minimize the cross modality distance
between the video and ground truth label, and make the network distinguish the
difference between real and pseudo modalities. The proposed framework can be
easily extended to other existing CTC based continuous SLR architectures.
Extensive experiments on two continuous SLR benchmarks, i.e.,
RWTH-PHOENIX-Weather and CSL, validate the effectiveness of our proposed
method.

提出了一种基于连接时序分类 (CTC) 目标函数和交叉模态增强的新型架构，来提高连续手语识别系统的性能，通过实现词频错误率的计算过程实现跨模态数据的增强和伪数据生成，有效地拓宽了数据集，提升了预测准确率。