Automatic speech recognition (ASR) techniques have become powerful tools,
enhancing efficiency in law enforcement scenarios. To ensure fairness for
demographic groups in different acoustic environments, ASR engines must be
tested across a variety of speakers in realistic settings. However, describing
the fairness discrepancies between models with confidence remains a challenge.
Meanwhile, most public ASR datasets are insufficient to perform a satisfying
fairness evaluation. To address the limitations, we built FairLENS - a
systematic fairness evaluation framework. We propose a novel and adaptable
evaluation method to examine the fairness disparity between different models.
We also collected a fairness evaluation dataset covering multiple scenarios and
demographic dimensions. Leveraging this framework, we conducted fairness
assessments on 1 open-source and 11 commercially available state-of-the-art ASR
models. Our results reveal that certain models exhibit more biases than others,
serving as a fairness guideline for users to make informed choices when
selecting ASR models for a given real-world scenario. We further explored model
biases towards specific demographic groups and observed that shifts in the
acoustic domain can lead to the emergence of new biases.

为了确保在不同的音响环境中为不同的人群提供公平性，我们建立了一个系统性的公平评估框架，通过一种新颖且适应性强的评估方法来检查不同模型之间的公平差距，并收集了多种情景和人口学维度的公平评估数据集。我们对 1 个开源和 11 个商业化最先进的 ASR 模型进行了公平评估，结果显示某些模型比其他模型存在更多偏见，这对用户在选择适合特定实际场景的 ASR 模型时提供了公平性指导。我们进一步探讨了模型对特定人口群体的偏见，并观察到音响领域的变化可能导致新的偏见出现。

公平 LENS：评估执法语音识别中的公平性

FairLENS: Assessing Fairness in Law Enforcement Speech Recognition

Acoustic environments affect acoustic characteristics of sound to be
recognized by physically interacting with sound wave propagation. Thus,
training acoustic models for audio and speech tasks requires regularization on
various acoustic environments in order to achieve robust performance in real
life applications. We propose FilterAugment, a data augmentation method for
regularization of acoustic models on various acoustic environments.
FilterAugment mimics acoustic filters by applying different weights on
frequency bands, therefore enables model to extract relevant information from
wider frequency region. It is an improved version of frequency masking which
masks information on random frequency bands. FilterAugment improved sound event
detection (SED) model performance by 6.50% while frequency masking only
improved 2.13% in terms of polyphonic sound detection score (PSDS). It achieved
equal error rate (EER) of 1.22% when applied to a text-independent speaker
verification model, outperforming model used frequency masking with EER of
1.26%. Prototype of FilterAugment was applied in our participation in DCASE
2021 challenge task 4, and played a major role in achieving the 3rd rank.

提出了一种名为 FilterAugment 的数据增强方法，用于在不同声学环境下正则化音频模型。通过在频率带上应用不同的权重，这种方法可以模拟声学滤波器，从而使模型能够从更广泛的频率区域提取相关信息。实验证明，与频率屏蔽相比，FilterAugment 在声音事件检测性能方面的提升为 6.50％，在说话人验证方面取得了 1.22％的等误率。