Subtitles appear on screen as short pieces of text, segmented based on formal
constraints (length) and syntactic/semantic criteria. Subtitle segmentation can
be evaluated with sequence segmentation metrics against a human reference.
However, standard segmentation metrics cannot be applied when systems generate
outputs different than the reference, e.g. with end-to-end subtitling systems.
In this paper, we study ways to conduct reference-based evaluations of
segmentation accuracy irrespective of the textual content. We first conduct a
systematic analysis of existing metrics for evaluating subtitle segmentation.
We then introduce $Sigma$, a new Subtitle Segmentation Score derived from an
approximate upper-bound of BLEU on segmentation boundaries, which allows us to
disentangle the effect of good segmentation from text quality. To compare
$Sigma$ with existing metrics, we further propose a boundary projection method
from imperfect hypotheses to the true reference. Results show that all metrics
are able to reward high quality output but for similar outputs system ranking
depends on each metric's sensitivity to error type. Our thorough analyses
suggest $Sigma$ is a promising segmentation candidate but its reliability over
other segmentation metrics remains to be validated through correlations with
human judgements.

研究字幕分割在使用不同于参考文本的输出时的评估方法，提出一种基于边界上的 BLEU 得分的字幕分割得分 $Sigma$。通过边界投影方法，将输出错误的假设映射到真实参考上，比较已有指标和 $Sigma$。研究结果表明，所有指标都能奖励高质量输出，但系统排名取决于每个指标对错误类型的敏感度，$Sigma$ 是一种有前途的分割候选者，但它与其他分割指标的可靠性仍需通过与人类判断相关性的验证来证实。