Understanding commonsense knowledge is crucial in the field of Natural
Language Processing (NLP). However, the presence of demographic terms in
commonsense knowledge poses a potential risk of compromising the performance of
NLP models. This study aims to investigate and propose methods for enhancing
the performance and effectiveness of a commonsense polarization classifier by
mitigating the influence of demographic terms. Three methods are introduced in
this paper: (1) hierarchical generalization of demographic terms (2)
threshold-based augmentation and (3) integration of hierarchical generalization
and threshold-based augmentation methods (IHTA). The first method involves
replacing demographic terms with more general ones based on a term hierarchy
ontology, aiming to mitigate the influence of specific terms. To address the
limited bias-related information, the second method measures the polarization
of demographic terms by comparing the changes in the model's predictions when
these terms are masked versus unmasked. This method augments commonsense
sentences containing terms with high polarization values by replacing their
predicates with synonyms generated by ChatGPT. The third method combines the
two approaches, starting with threshold-based augmentation followed by
hierarchical generalization. The experiments show that the first method
increases the accuracy over the baseline by 2.33%, and the second one by 0.96%
over standard augmentation methods. The IHTA techniques yielded an 8.82% and
9.96% higher accuracy than threshold-based and standard augmentation methods,
respectively.

通过层次概括、基于阈值的增强和层次概括与阈值增强方法相结合等三种方法，提高了常识极化分类器的性能和效果，减轻了人口统计术语对 NLP 模型性能的影响。

通过减少人口统计术语的影响来提升常识偏见分类

Improving Commonsense Bias Classification by Mitigating the Influence of  Demographic Terms

Transformers trained on natural language data have been shown to learn its
hierarchical structure and generalize to sentences with unseen syntactic
structures without explicitly encoding any structural bias. In this work, we
investigate sources of inductive bias in transformer models and their training
that could cause such generalization behavior to emerge. We extensively
experiment with transformer models trained on multiple synthetic datasets and
with different training objectives and show that while other objectives e.g.
sequence-to-sequence modeling, prefix language modeling, often failed to lead
to hierarchical generalization, models trained with the language modeling
objective consistently learned to generalize hierarchically. We then conduct
pruning experiments to study how transformers trained with the language
modeling objective encode hierarchical structure. When pruned, we find joint
existence of subnetworks within the model with different generalization
behaviors (subnetworks corresponding to hierarchical structure and linear
order). Finally, we take a Bayesian perspective to further uncover
transformers' preference for hierarchical generalization: We establish a
correlation between whether transformers generalize hierarchically on a dataset
and whether the simplest explanation of that dataset is provided by a
hierarchical grammar compared to regular grammars exhibiting linear
generalization.

透过对自然语言数据训练的 Transformer 模型的研究和实验证明，具备语言建模目标的 Transformer 模型更容易学习和推广层次结构，并在处理无法预见的句法结构的句子时表现优异。