We argue that we need to evaluate model interpretability methods 'in the
wild', i.e., in situations where professionals make critical decisions, and
models can potentially assist them. We present an in-the-wild evaluation of
token attribution based on Deep Taylor Decomposition, with professional
journalists performing reliability assessments. We find that using this method
in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to
faster and better human decision-making, as well as a more critical attitude
toward news sources among the journalists. We present a comparison of human and
model rationales, as well as a qualitative analysis of the journalists'
experiences with machine-in-the-loop decision making.

本文通过在野外环境下的评估，研究了基于 Deep Taylor Decomposition 的标记归因模型，并发现其与经过 Gossip Corpus 微调的 RoBERTa-Large 协同使用可以提高专业记者的决策速度和决策质量，在决策过程中提高了对新闻来源的批判性。本文还进行了人类和模型理性的比较研究，并对记者们在机器辅助下决策过程的体验进行了定性分析。

评估 Deep Taylor 分解在野外的可靠性评估中的应用

Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild

Transfer learning techniques are particularly useful in NLP tasks where a
sizable amount of high-quality annotated data is difficult to obtain. Current
approaches directly adapt a pre-trained language model (LM) on in-domain text
before fine-tuning to downstream tasks. We show that extending the vocabulary
of the LM with domain-specific terms leads to further gains. To a bigger
effect, we utilize structure in the unlabeled data to create auxiliary
synthetic tasks, which helps the LM transfer to downstream tasks. We apply
these approaches incrementally on a pre-trained Roberta-large LM and show
considerable performance gain on three tasks in the IT domain: Extractive
Reading Comprehension, Document Ranking and Duplicate Question Detection.

本文研究使用预训练语言模型的转移学习、领域特定术语扩展词汇表以及利用未标记数据结构创造辅助合成任务等方法，在 IT 领域的三个任务中，采用逐步应用的策略在预先训练的 Roberta-large LM 上显示出显着的性能提升。

低资源领域适应的多阶段预训练

Multi-Stage Pre-training for Low-Resource Domain Adaptation

A critical part of reading is being able to understand the temporal
relationships between events described in a passage of text, even when those
relationships are not explicitly stated. However, current machine reading
comprehension benchmarks have practically no questions that test temporal
phenomena, so systems trained on these benchmarks have no capacity to answer
questions such as "what happened before/after [some event]?" We introduce
TORQUE, a new English reading comprehension benchmark built on 3.2k news
snippets with 21k human-generated questions querying temporal relationships.
Results show that RoBERTa-large achieves an exact-match score of 51% on the
test set of TORQUE, about 30% behind human performance.

本研究介绍了 TORQUE，这是一个新的英语阅读理解基准，用于检测时间关系，结果显示 RoBERTa-large 在 TORQUE 的测试集上的精确匹配分数为 51％，约为人类表现的 30％。

TORQUE: 一个包含时间序问题的阅读理解数据集

TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

We present a model to predict fine-grained emotions along the continuous
dimensions of valence, arousal, and dominance (VAD) with a corpus with
categorical emotion annotations. Our model is trained by minimizing the EMD
(Earth Mover's Distance) loss between the predicted VAD score distribution and
the categorical emotion distributions sorted along VAD, and it can
simultaneously classify the emotion categories and predict the VAD scores for a
given sentence. We use pre-trained RoBERTa-Large and fine-tune on three
different corpora with categorical labels and evaluate on EmoBank corpus with
VAD scores. We show that our approach reaches comparable performance to that of
the state-of-the-art classifiers in categorical emotion classification and
shows significant positive correlations with the ground truth VAD scores. Also,
further training with supervision of VAD labels leads to improved performance
especially when dataset is small. We also present examples of predictions of
appropriate emotion words that are not part of the original annotations.

本文介绍一种使用 RoBERTa-Large 进行监督学习，通过最小化预测 VAD 得分分布与分类情感分布之间的 EMD 损失，同时分类情感类别和预测 VAD 得分的方法，其在情感分类和与地面真值 VAD 得分之间具有显著的正相关性，并利用情感数据进行训练和预测。