Speech-to-Speech and Speech-to-Text translation are currently dynamic areas
of research. To contribute to these fields, we present SpeechAlign, a framework
to evaluate the underexplored field of source-target alignment in speech
models. Our framework has two core components. First, to tackle the absence of
suitable evaluation datasets, we introduce the Speech Gold Alignment dataset,
built upon a English-German text translation gold alignment dataset. Secondly,
we introduce two novel metrics, Speech Alignment Error Rate (SAER) and
Time-weighted Speech Alignment Error Rate (TW-SAER), to evaluate alignment
quality in speech models. By publishing SpeechAlign we provide an accessible
evaluation framework for model assessment, and we employ it to benchmark
open-source Speech Translation models.

SpeechAlign 是一个框架，用于评估语音模型中源 - 目标对齐领域的尚未充分探索的领域，并提供了两个新的度量标准来评估语音模型的对齐质量，通过发布 SpeechAlign 来为模型评估提供一个可访问的评估框架，并用它来评估开源的语音翻译模型。

SpeechAlign：语音翻译对齐评估框架

SpeechAlign: a Framework for Speech Translation Alignment Evaluation

Non-autoregressive (NAR) models generate all the tokens of a sequence in
parallel, resulting in faster generation speed compared to their autoregressive
(AR) counterparts but at the cost of lower accuracy. Different techniques
including knowledge distillation and source-target alignment have been proposed
to bridge the gap between AR and NAR models in various tasks such as neural
machine translation (NMT), automatic speech recognition (ASR), and text to
speech (TTS). With the help of those techniques, NAR models can catch up with
the accuracy of AR models in some tasks but not in some others. In this work,
we conduct a study to understand the difficulty of NAR sequence generation and
try to answer: (1) Why NAR models can catch up with AR models in some tasks but
not all? (2) Why techniques like knowledge distillation and source-target
alignment can help NAR models. Since the main difference between AR and NAR
models is that NAR models do not use dependency among target tokens while AR
models do, intuitively the difficulty of NAR sequence generation heavily
depends on the strongness of dependency among target tokens. To quantify such
dependency, we propose an analysis model called CoMMA to characterize the
difficulty of different NAR sequence generation tasks. We have several
interesting findings: 1) Among the NMT, ASR and TTS tasks, ASR has the most
target-token dependency while TTS has the least. 2) Knowledge distillation
reduces the target-token dependency in target sequence and thus improves the
accuracy of NAR models. 3) Source-target alignment constraint encourages
dependency of a target token on source tokens and thus eases the training of
NAR models.

该研究通过提出一种分析模型来表征 Non-autoregressive 模型生成不同序列任务的难易程度，并发现知识蒸馏和源 - 目标对齐技术可以帮助提升 Non-autoregressive 模型的准确性，减少目标序列的依赖性。