Recent works demonstrate that voice assistants do not perform equally well
for everyone, but research on demographic robustness of speech technologies is
still scarce. This is mainly due to the rarity of large datasets with
controlled demographic tags. This paper introduces the Sonos Voice Control Bias
Assessment Dataset, an open dataset composed of voice assistant requests for
North American English in the music domain (1,038 speakers, 166 hours, 170k
audio samples, with 9,040 unique labelled transcripts) with a controlled
demographic diversity (gender, age, dialectal region and ethnicity). We also
release a statistical demographic bias assessment methodology, at the
univariate and multivariate levels, tailored to this specific use case and
leveraging spoken language understanding metrics rather than transcription
accuracy, which we believe is a better proxy for user experience. To
demonstrate the capabilities of this dataset and statistical method to detect
demographic bias, we consider a pair of state-of-the-art Automatic Speech
Recognition and Spoken Language Understanding models. Results show
statistically significant differences in performance across age, dialectal
region and ethnicity. Multivariate tests are crucial to shed light on mixed
effects between dialectal region, gender and age.

该研究引入了 Sonos Voice Control Bias Assessment Dataset，一个包含控制人口统计学标签的大型数据集，通过针对音乐领域的语音助手请求（1,038 名说话者，166 小时，170k 音频样本，9,040 个唯一标记的转录文本）进行了控制人口统计多样性（性别、年龄、方言区域和种族）。研究还提出了一种统计的人口统计偏见评估方法，适用于这种特定的用例，并利用口语理解度量标准而非转录准确性，我们认为后者更能代表用户体验。通过考虑一对最先进的自动语音识别和口语理解模型，展示了该数据集和统计方法检测人口统计偏见的能力。结果表明在年龄、方言区域和种族之间性能存在统计上显著差异。多变量测试对于阐明方言区域、性别和年龄之间的混合效应至关重要。