We present a robust methodology for evaluating biases in natural language
generation(NLG) systems. Previous works use fixed hand-crafted prefix templates
with mentions of various demographic groups to prompt models to generate
continuations for bias analysis. These fixed prefix templates could themselves
be specific in terms of styles or linguistic structures, which may lead to
unreliable fairness conclusions that are not representative of the general
trends from tone varying prompts. To study this problem, we paraphrase the
prompts with different syntactic structures and use these to evaluate
demographic bias in NLG systems. Our results suggest similar overall bias
trends but some syntactic structures lead to contradictory conclusions compared
to past works. We show that our methodology is more robust and that some
syntactic structures prompt more toxic content while others could prompt less
biased generation. This suggests the importance of not relying on a fixed
syntactic structure and using tone-invariant prompts. Introducing
syntactically-diverse prompts can achieve more robust NLG (bias) evaluation.

通过使用不同的语法结构，本文提出了一种鲁棒的自然语言生成系统偏见评估方法，其结果显示采用语法多样性的提示可以实现更鲁棒的 NLG（偏见）评估。