Natural language generation tools are powerful and effective for generating
content. However, language models are known to display bias and fairness
issues, making them impractical to deploy for many use cases. We here focus on
how fairness issues impact automatically generated test content, which can have
stringent requirements to ensure the test measures only what it was intended to
measure. Specifically, we identify test content that is focused on particular
domains and experiences that only reflect a certain demographic or that are
potentially emotionally upsetting; both of which could inadvertently impact a
test-taker's score. This kind of content doesn't reflect typical biases out of
context, making it challenging even for modern models that contain safeguards.
We build a dataset of 621 generated texts annotated for fairness and explore a
variety of methods for classification: fine-tuning, topic-based classification,
and prompting, including few-shot and self-correcting prompts. We find that
combining prompt self-correction and few-shot learning performs best, yielding
an F1 score of .791 on our held-out test set, while much smaller BERT- and
topic-based models have competitive performance on out-of-domain data.

旨在探讨自然语言生成中的公平问题，特别关注自动生成的测试内容对测试结果的影响，构建数据集并使用多种分类方法进行实验，发现自我纠正和少样本学习相结合的方法在公平性上有最佳表现。