This paper presents a partial reproduction of Generating Fact Checking
Explanations by Anatanasova et al (2020) as part of the ReproHum element of the
ReproNLP shared task to reproduce the findings of NLP research regarding human
evaluation. This shared task aims to investigate the extent to which NLP as a
field is becoming more or less reproducible over time. Following the
instructions provided by the task organisers and the original authors, we
collect relative rankings of 3 fact-checking explanations (comprising a gold
standard and the outputs of 2 models) for 40 inputs on the criteria of
Coverage. The results of our reproduction and reanalysis of the original work's
raw results lend support to the original findings, with similar patterns seen
between the original work and our reproduction. Whilst we observe slight
variation from the original results, our findings support the main conclusions
drawn by the original authors pertaining to the efficacy of their proposed
models.

这篇论文通过 ReproHum 元素，部分复现了 Anatanasova 等人（2020）的《生成事实检查解释》。在 ReproNLP 共同任务中，本共享任务旨在研究 NLP 作为一个领域在时间上变得更加可复现的程度。通过根据任务组织者和原始作者提供的指南，我们对 40 个输入的 3 个事实检查解释（包括一个黄金标准和两个模型的输出）进行了相对排名，评估了其覆盖范围。我们的复现和对原始结果的再分析结果支持原论文的发现，原始论文与我们的复现结果之间存在类似的模式。尽管我们观察到与原始结果略有不同的变化，但我们的发现支持原始作者关于其提出的模型的有效性的主要结论。

ReproHum #0087-01: 生成事实检查解释的人工评估复现报告

ReproHum #0087-01: Human Evaluation Reproduction Report for Generating  Fact Checking Explanations

In this work, we introduce X-FACT: the largest publicly available
multilingual dataset for factual verification of naturally existing real-world
claims. The dataset contains short statements in 25 languages and is labeled
for veracity by expert fact-checkers. The dataset includes a multilingual
evaluation benchmark that measures both out-of-domain generalization, and
zero-shot capabilities of the multilingual models. Using state-of-the-art
multilingual transformer-based models, we develop several automated
fact-checking models that, along with textual claims, make use of additional
metadata and evidence from news stories retrieved using a search engine.
Empirically, our best model attains an F-score of around 40%, suggesting that
our dataset is a challenging benchmark for evaluation of multilingual
fact-checking models.

本研究介绍了 X-FACT 数据集，这是目前最大的公开的多语言事实验证数据集，其中包含 25 种语言的短语句，并由专业事实检查者标记真实性，数据集包含多语言评估基准测试，并使用现代化的多语言变压器模型开发了多个自动事实检查模型，实证表明模型的最佳性能 F1 分数约为 40％，因此是评估多语言事实检查模型的挑战性基准测试。

X-FACT：一个新的用于多语言事实检查的基准数据集

X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

This paper describes a baseline for the second iteration of the Fact
Extraction and VERification shared task (FEVER2.0) which explores the
resilience of systems through adversarial evaluation. We present a collection
of simple adversarial attacks against systems that participated in the first
FEVER shared task. FEVER modeled the assessment of truthfulness of written
claims as a joint information retrieval and natural language inference task
using evidence from Wikipedia. A large number of participants made use of deep
neural networks in their submissions to the shared task. The extent as to
whether such models understand language has been the subject of a number of
recent investigations and discussion in literature. In this paper, we present a
simple method of generating entailment-preserving and entailment-altering
perturbations of instances by common patterns within the training data. We find
that a number of systems are greatly affected with absolute losses in
classification accuracy of up to $29\%$ on the newly perturbed instances. Using
these newly generated instances, we construct a sample submission for the
FEVER2.0 shared task. Addressing these types of attacks will aid in building
more robust fact-checking models, as well as suggest directions to expand the
datasets.

本文的 FEVER2.0 基准测试通过生成针对系统的对抗性攻击探索了 NN 模型在事实提取和验证任务中的鲁棒性，结果表明深度神经网络在处理 pattren 异常的样本时效果较差，提出对这类攻击进行研究有助于构建更健壮的事实检查模型，同时建议扩展数据集。