Fairness has been a critical issue that affects the adoption of deep learning
models in real practice. To improve model fairness, many existing methods have
been proposed and evaluated to be effective in their own contexts. However,
there is still no systematic evaluation among them for a comprehensive
comparison under the same context, which makes it hard to understand the
performance distinction among them, hindering the research progress and
practical adoption of them. To fill this gap, this paper endeavours to conduct
the first large-scale empirical study to comprehensively compare the
performance of existing state-of-the-art fairness improving techniques.
Specifically, we target the widely-used application scenario of image
classification, and utilized three different datasets and five commonly-used
performance metrics to assess in total 13 methods from diverse categories. Our
findings reveal substantial variations in the performance of each method across
different datasets and sensitive attributes, indicating over-fitting on
specific datasets by many existing methods. Furthermore, different fairness
evaluation metrics, due to their distinct focuses, yield significantly
different assessment results. Overall, we observe that pre-processing methods
and in-processing methods outperform post-processing methods, with
pre-processing methods exhibiting the best performance. Our empirical study
offers comprehensive recommendations for enhancing fairness in deep learning
models. We approach the problem from multiple dimensions, aiming to provide a
uniform evaluation platform and inspire researchers to explore more effective
fairness solutions via a set of implications.

本论文开展了第一次大规模实证研究，全面比较了现有最先进的公平性改进技术的性能，结果显示不同方法在不同数据集和敏感属性上表现存在较大差异，且不同的公平评估指标产生显著不同的评估结果，其中预处理方法和内处理方法优于后处理方法，预处理方法表现最佳，本研究为深度学习模型公平性提供了全面的建议。