Understanding the importance of the inputs on the output is useful across
many tasks. This work provides an information-theoretic framework to analyse
the influence of inputs for text classification tasks. Natural language
processing (NLP) tasks take either a single element input or multiple element
inputs to predict an output variable, where an element is a block of text. Each
text element has two components: an associated semantic meaning and a
linguistic realization. Multiple-choice reading comprehension (MCRC) and
sentiment classification (SC) are selected to showcase the framework. For MCRC,
it is found that the context influence on the output compared to the question
influence reduces on more challenging datasets. In particular, more challenging
contexts allow a greater variation in complexity of questions. Hence, test
creators need to carefully consider the choice of the context when designing
multiple-choice questions for assessment. For SC, it is found the semantic
meaning of the input text dominates (above 80\% for all datasets considered)
compared to its linguistic realisation when determining the sentiment. The
framework is made available at:
this https URL

该研究提供了一个信息理论框架，用于分析文本分类任务中输入对输出的影响，研究发现多选阅读理解任务中，上下文与问题对输出的影响随着数据集的挑战程度增加而减少；而情感分类任务中，输入文本的语义含义在决定情感时比其语言表达更为重要。

NLP 分类任务的信息论分析方法

An Information-Theoretic Approach to Analyze NLP Classification Tasks

Multiple-Choice Reading Comprehension (MCRC) requires the model to read the
passage and question, and select the correct answer among the given options.
Recent state-of-the-art models have achieved impressive performance on multiple
MCRC datasets. However, such performance may not reflect the model's true
ability of language understanding and reasoning. In this work, we adopt two
approaches to investigate what BERT learns from MCRC datasets: 1) an
un-readable data attack, in which we add keywords to confuse BERT, leading to a
significant performance drop; and 2) an un-answerable data training, in which
we train BERT on partial or shuffled input. Under un-answerable data training,
BERT achieves unexpectedly high performance. Based on our experiments on the 5
key MCRC datasets - RACE, MCTest, MCScript, MCScript2.0, DREAM - we observe
that 1) fine-tuned BERT mainly learns how keywords lead to correct prediction,
instead of learning semantic understanding and reasoning; and 2) BERT does not
need correct syntactic information to solve the task; 3) there exists artifacts
in these datasets such that they can be solved even without the full context.

通过对多选阅读理解数据集进行实验，本文发现：1）BERT 模型主要是依靠关键字来进行答案预测，而不是通过语义理解和推理；2）BERT 模型不需要正确的句法信息来解决任务；3）这些数据集中存在着一些人工痕迹，即使没有完整上下文也可以解决任务。

BERT 从多项选择阅读理解数据中学到了什么？

What does BERT Learn from Multiple-Choice Reading Comprehension  Datasets?

Multiple-choice reading comprehension (MCRC) is the task of selecting the
correct answer from multiple options given a question and an article. Existing
MCRC models typically either read each option independently or compute a
fixed-length representation for each option before comparing them. However,
humans typically compare the options at multiple-granularity level before
reading the article in detail to make reasoning more efficient. Mimicking
humans, we propose an option comparison network (OCN) for MCRC which compares
options at word-level to better identify their correlations to help reasoning.
Specially, each option is encoded into a vector sequence using a skimmer to
retain fine-grained information as much as possible. An attention mechanism is
leveraged to compare these sequences vector-by-vector to identify more subtle
correlations between options, which is potentially valuable for reasoning.
Experimental results on the human English exam MCRC dataset RACE show that our
model outperforms existing methods significantly. Moreover, it is also the
first model that surpasses Amazon Mechanical Turker performance on the whole
dataset.

本文研究了多选阅读理解中，通过词级别对选项之间的关系进行比较获得更好的推理能力的方法，并在 RACE 数据集中的表现超过其他模型，成为第一个超过 Amazon Mechanical Turk 表现的模型。