Most existing multi-hop datasets are extractive answer datasets, where the
answers to the questions can be extracted directly from the provided context.
This often leads models to use heuristics or shortcuts instead of performing
true multi-hop reasoning. In this paper, we propose a new multi-hop dataset,
MoreHopQA, which shifts from extractive to generative answers. Our dataset is
created by utilizing three existing multi-hop datasets: HotpotQA,
2WikiMultihopQA, and MuSiQue. Instead of relying solely on factual reasoning,
we enhance the existing multi-hop questions by adding another layer of
questioning that involves one, two, or all three of the following types of
reasoning: commonsense, arithmetic, and symbolic. Our dataset is created
through a semi-automated process, resulting in a dataset with 1,118 samples
that have undergone human verification. We then use our dataset to evaluate
five different large language models: Mistral 7B, Gemma 7B, Llama 3 (8B and
70B), and GPT-4. We also design various cases to analyze the reasoning steps in
the question-answering process. Our results show that models perform well on
initial multi-hop questions but struggle with our extended questions,
indicating that our dataset is more challenging than previous ones. Our
analysis of question decomposition reveals that although models can correctly
answer questions, only a portion - 38.7% for GPT-4 and 33.4% for Llama3-70B -
achieve perfect reasoning, where all corresponding sub-questions are answered
correctly. Evaluation code and data are available at
this https URL

将现有的多跳数据集从抽取性答案转变为生成性答案，通过添加常识、算术和符号推理等问题层次，我们创建了一个新的多跳数据集 MoreHopQA，以评估五个大型语言模型，并发现相较于以往的数据集，我们的数据集更具挑战性，其中部分问题分解的分析表明虽然模型可以正确回答问题，但只有某个部分实现了完美推理。

MoreHopQA: 超越多跳推理

MoreHopQA: More Than Multi-hop Reasoning

For a natural language understanding benchmark to be useful in research, it
has to consist of examples that are diverse and difficult enough to
discriminate among current and near-future state-of-the-art systems. However,
we do not yet know how best to select text sources to collect a variety of
challenging examples. In this study, we crowdsource multiple-choice reading
comprehension questions for passages taken from seven qualitatively distinct
sources, analyzing what attributes of passages contribute to the difficulty and
question types of the collected examples. To our surprise, we find that passage
source, length, and readability measures do not significantly affect question
difficulty. Through our manual annotation of seven reasoning types, we observe
several trends between passage sources and reasoning types, e.g., logical
reasoning is more often required in questions written for technical passages.
These results suggest that when creating a new benchmark dataset, selecting a
diverse set of passages can help ensure a diverse range of question types, but
that passage difficulty need not be a priority.

通过众包多选阅读理解问题，分析来自七个质量不同的来源的段落的属性如何影响问题难度及类型，发现段落的来源、长度和可读性等因素并不影响问题的难度，而逻辑推理更常常出现在针对技术文章的问题中。因此，在创建新的基准数据集时，选择多样性的段落可以确保问题类型的多样性，而段落的难度则不是最重要的。

阅读理解问题为何难？

What Makes Reading Comprehension Questions Difficult?

The recent work of Clark et al. introduces the AI2 Reasoning Challenge (ARC)
and the associated ARC dataset that partitions open domain, complex science
questions into an Easy Set and a Challenge Set. That paper includes an analysis
of 100 questions with respect to the types of knowledge and reasoning required
to answer them; however, it does not include clear definitions of these types,
nor does it offer information about the quality of the labels. We propose a
comprehensive set of definitions of knowledge and reasoning types necessary for
answering the questions in the ARC dataset. Using ten annotators and a
sophisticated annotation interface, we analyze the distribution of labels
across the Challenge Set and statistics related to them. Additionally, we
demonstrate that although naive information retrieval methods return sentences
that are irrelevant to answering the query, sufficient supporting text is often
present in the (ARC) corpus. Evaluating with human-selected relevant sentences
improves the performance of a neural machine comprehension model by 42 points.

本文提出了一套完整的科学问题解答所需的知识和推理类型定义方法，回答 ARC 数据集中的问题需要的标注信息分布和相关统计的分析，证明了人工选取的相关句子能够提高神经机器理解模型的 42 个百分点的性能。