The popularity of automatic speech recognition (ASR) systems nowadays leads
to an increasing need for improving their accessibility. Handling stuttering
speech is an important feature for accessible ASR systems. To improve the
accessibility of ASR systems for stutterers, we need to expose and analyze the
failures of ASR systems on stuttering speech. The speech datasets recorded from
stutterers are not diverse enough to expose most of the failures. Furthermore,
these datasets lack ground truth information about the non-stuttered text,
rendering them unsuitable as comprehensive test suites. Therefore, a
methodology for generating stuttering speech as test inputs to test and analyze
the performance of ASR systems is needed. However, generating valid test inputs
in this scenario is challenging. The reason is that although the generated test
inputs should mimic how stutterers speak, they should also be diverse enough to
trigger more failures. To address the challenge, we propose ASTER, a technique
for automatically testing the accessibility of ASR systems. ASTER can generate
valid test cases by injecting five different types of stuttering. The generated
test cases can both simulate realistic stuttering speech and expose failures in
ASR systems. Moreover, ASTER can further enhance the quality of the test cases
with a multi-objective optimization-based seed updating algorithm. We
implemented ASTER as a framework and evaluated it on four open-source ASR
models and three commercial ASR systems. We conduct a comprehensive evaluation
of ASTER and find that it significantly increases the word error rate, match
error rate, and word information loss in the evaluated ASR systems.
Additionally, our user study demonstrates that the generated stuttering audio
is indistinguishable from real-world stuttering audio clips.

提供一种用于测试和分析自动语音识别系统性能的测试用例生成方法，该方法通过模拟真实的口吃语音并注入多种不同类型的口吃来生成有效的测试用例，并在评估中明显增加了评估的 ASR 系统的词误差率、匹配错误率和词信息损失。