In fairness audits, a standard objective is to detect whether a given
algorithm performs substantially differently between subgroups. Properly
powering the statistical analysis of such audits is crucial for obtaining
informative fairness assessments, as it ensures a high probability of detecting
unfairness when it exists. However, limited guidance is available on the amount
of data necessary for a fairness audit, lacking directly applicable results
concerning commonly used fairness metrics. Additionally, the consideration of
unequal subgroup sample sizes is also missing. In this tutorial, we address
these issues by providing guidance on how to determine the required subgroup
sample sizes to maximize the statistical power of hypothesis tests for
detecting unfairness. Our findings are applicable to audits of binary
classification models and multiple fairness metrics derived as summaries of the
confusion matrix. Furthermore, we discuss other aspects of audit study designs
that can increase the reliability of audit results.

公平审计中，确定所需子群样本量以最大化假设检验的统计功效对于获得信息丰富的公平评估至关重要，本文解决了这些问题并提供了相关指导，适用于二元分类模型的审计和多种公平度量方法的总结。此外，文章还讨论了其他可提高审计结果可靠性的研究方法设计方面的问题。