Language models have demonstrated strong performance on various natural
language understanding tasks. Similar to humans, language models could also
have their own bias that is learned from the training data. As more and more
downstream tasks integrate language models as part of the pipeline, it is
necessary to understand the internal stereotypical representation and the
methods to mitigate the negative effects. In this paper, we proposed a simple
method to test the internal stereotypical representation in pre-trained
language models using counterexamples. We mainly focused on gender bias, but
the method can be extended to other types of bias. We evaluated models on 9
different cloze-style prompts consisting of knowledge and base prompts. Our
results indicate that pre-trained language models show a certain amount of
robustness when using unrelated knowledge, and prefer shallow linguistic cues,
such as word position and syntactic structure, to alter the internal
stereotypical representation. Such findings shed light on how to manipulate
language models in a neutral approach for both finetuning and evaluation.

本文研究语言模型中的偏见问题，提出了一种测试内部陈规陈瘤表现的简单方法，并探讨了该方法在性别偏见方面的应用。实验表明，使用无关的知识时，预训练的语言模型表现出一定的鲁棒性，并倾向于选择词语位置和句法结构等浅层语言线索来改变内部陈规陈瘤表现，从而为微调和评估中如何中性地处理语言模型提供了新的思路。

测试预训练语言模型中的刻板印象

Counteracts: Testing Stereotypical Representation in Pre-trained Language Models

We present TacoBot, a task-oriented dialogue system built for the inaugural
Alexa Prize TaskBot Challenge, which assists users in completing multi-step
cooking and home improvement tasks. TacoBot is designed with a user-centered
principle and aspires to deliver a collaborative and accessible dialogue
experience. Towards that end, it is equipped with accurate language
understanding, flexible dialogue management, and engaging response generation.
Furthermore, TacoBot is backed by a strong search engine and an automated
end-to-end test suite. In bootstrapping the development of TacoBot, we explore
a series of data augmentation strategies to train advanced neural language
processing models and continuously improve the dialogue experience with
collected real conversations. At the end of the semifinals, TacoBot achieved an
average rating of 3.55/5.0.

TacoBot 是一个任务导向的对话系统，旨在协助用户完成多步骤的烹饪和家庭维护任务，它具备准确的语言理解、灵活的对话管理和引人入胜的响应生成，通过各种数据增强策略来训练先进的神经语言处理模型并持续改进对话体验。在半决赛结束时，TacoBot 的平均评分为 3.55/5.0。

基于用户和任务的对话系统引导

Bootstrapping a User-Centered Task-Oriented Dialogue System

The differences in written text and conversational speech are substantial;
previous parsers trained on treebanked text have given very poor results on
spontaneous speech. For spoken language, the mismatch in style also extends to
prosodic cues, though it is less well understood. This paper re-examines the
use of written text in parsing speech in the context of recent advances in
neural language processing. We show that neural approaches facilitate using
written text to improve parsing of spontaneous speech, and that prosody further
improves over this state-of-the-art result. Further, we find an asymmetric
degradation from read vs. spontaneous mismatch, with spontaneous speech more
generally useful for training parsers.

本文通过深度学习技术重新审视了语言处理中书面文本对口语处理的作用，发现书面文本与口语数据的不匹配使得基于书面文本训练的处理器应用于口语处理时表现不佳，而基于神经网络的方法使书面文本可以用于提高口语处理性能，同时考虑语调因素可以进一步提高性能，特别是对于模型训练而言，自然语言生成的语音数据更具有实际应用价值。