Sign languages are multi-channel languages that communicate information through not just the hands (manual signals) but also facial expressions and upper body movements (non-manual signals). However, since automatic sign language translation is usually performed by generating a single sequence of glosses, researchers eschew non-manual and co-occurring manual signals in favor of a simplified list of manual glosses. This can lead to significant information loss and ambiguity. In this paper, we introduce a new task named multi-channel sign language translation (MCSLT) and present a novel metric, SignBLEU, designed to capture multiple signal channels. We validated SignBLEU on a system-level task using three sign language corpora with varied linguistic structures and transcription methodologies and examined its correlation with human judgment through two segment-level tasks. We found that SignBLEU consistently correlates better with human judgment than competing metrics. To facilitate further MCSLT research, we report benchmark scores for the three sign language corpora and release the source code for SignBLEU at https://github.com/eq4all-projects/SignBLEU.

该研究介绍了一种名为多信道手语翻译 (MCSLT) 的新任务，并提出了一种用于捕捉多种信号通道的新度量标准 SignBLEU。通过在三个手语文本语料库上验证 SignBLEU，并通过两个部分级任务考察其与人类判断的相关性，研究发现 SignBLEU 与人类判断的相关性始终优于竞争度量标准。为了促进进一步的 MCSLT 研究，研究报告了三个手语文本语料库的基准分数，并发布了 SignBLEU 的源代码。

SignBLEU：多通道手语翻译的自动评估