Data collected from the real world tends to be biased, unbalanced, and at
risk of exposing sensitive and private information. This reality has given rise
to the idea of creating synthetic datasets to alleviate risk, bias, harm, and
privacy concerns inherent in the real data. This concept relies on Generative
AI models to produce unbiased, privacy-preserving synthetic data while being
true to the real data. In this new paradigm, how can we tell if this approach
delivers on its promises? We present an auditing framework that offers a
holistic assessment of synthetic datasets and AI models trained on them,
centered around bias and discrimination prevention, fidelity to the real data,
utility, robustness, and privacy preservation. We showcase our framework by
auditing multiple generative models on diverse use cases, including education,
healthcare, banking, human resources, and across different modalities, from
tabular, to time-series, to natural language. Our use cases demonstrate the
importance of a holistic assessment in order to ensure compliance with
socio-technical safeguards that regulators and policymakers are increasingly
enforcing. For this purpose, we introduce the trust index that ranks multiple
synthetic datasets based on their prescribed safeguards and their desired
trade-offs. Moreover, we devise a trust-index-driven model selection and
cross-validation procedure via auditing in the training loop that we showcase
on a class of transformer models that we dub TrustFormers, across different
modalities. This trust-driven model selection allows for controllable trust
trade-offs in the resulting synthetic data. We instrument our auditing
framework with workflows that connect different stakeholders from model
development to audit and certification via a synthetic data auditing report.

本文提出一个审核框架，围绕偏见和歧视预防、与真实数据的一致性、效用、鲁棒性和隐私保护进行综合评估，并引入信任指数对多个合成数据集进行排名，通过审核和认证报告连接不同利益相关者。