Recent work on transformer-based neural networks has led to impressive
advances on multiple-choice natural language understanding (NLU) problems, such
as Question Answering (QA) and abductive reasoning. Despite these advances,
there is limited work still on understanding whether these models respond to
perturbed multiple-choice instances in a sufficiently robust manner that would
allow them to be trusted in real-world situations. We present four confusion
probes, inspired by similar phenomena first identified in the behavioral
science community, to test for problems such as prior bias and choice
paralysis. Experimentally, we probe a widely used transformer-based
multiple-choice NLU system using four established benchmark datasets. Here we
show that the model exhibits significant prior bias and to a lesser, but still
highly significant degree, choice paralysis, in addition to other problems. Our
results suggest that stronger testing protocols and additional benchmarks may
be necessary before the language models are used in front-facing systems or
decision making with real world consequences.

本文研究了基于 transformer 的神经网络在自然语言理解方面的多项选择问题，主要针对先验偏差和选择麻烦等问题做了四项混淆测试，并表明该模型在应对这些问题时表现出明显的偏差和选择麻烦，建议需要进一步增强测试协议和基准测试以确保语言模型在面向实际决策的系统中得到合理的应用。

通过四个实验探针理解基于 Transformer 的语言表示模型中的先验偏差和选择瘫痪

Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes

Scene Graph Generation (SGG) aims to build a structured representation of a
scene using objects and pairwise relationships, which benefits downstream
tasks. However, current SGG methods usually suffer from sub-optimal scene graph
generation because of the long-tailed distribution of training data. To address
this problem, we propose Resistance Training using Prior Bias (RTPB) for the
scene graph generation. Specifically, RTPB uses a distributed-based prior bias
to improve models' detecting ability on less frequent relationships during
training, thus improving the model generalizability on tail categories. In
addition, to further explore the contextual information of objects and
relationships, we design a contextual encoding backbone network, termed as Dual
Transformer (DTrans). We perform extensive experiments on a very popular
benchmark, VG150, to demonstrate the effectiveness of our method for the
unbiased scene graph generation. In specific, our RTPB achieves an improvement
of over 10% under the mean recall when applied to current SGG methods.
Furthermore, DTrans with RTPB outperforms nearly all state-of-the-art methods
with a large margin.

本研究提出了一种基于分布式先验偏差的坚韧训练方法 (RTPB) 用于解决目前场景图生成中训练数据分布的长尾问题，并通过验证实验在 VG150 数据集上表明了其有效性和性能提升。另外，我们还设计了一种称为 Dual Transformer (DTrans) 的上下文编码骨干网络，进一步探测对象和关系的上下文信息，含有 RTPB 的 DTrans 方法性能较之现有首屈一指的方法都有很大的提升。