When provided with sufficient explanatory context, smaller Language Models
have been shown to exhibit strong reasoning ability on challenging short-answer
question-answering tasks where the questions are unseen in training. We
evaluate two methods for further improvement in this setting. Both methods
focus on combining rationales generated by a larger Language Model with longer
contexts created from a multi-hop dense retrieval system. The first method
($\textit{RR}$) involves training a Rationale Ranking model to score both
generated rationales and retrieved contexts with respect to relevance and
truthfulness. We then use the scores to derive combined contexts from both
knowledge sources using a number of combinatory strategies. For the second
method ($\textit{RATD}$) we train a smaller Reasoning model using
retrieval-augmented training datasets such that it becomes proficient at
utilising relevant information from longer text sequences that may be only
partially evidential and frequently contain many irrelevant sentences.
Generally we find that both methods are effective but that the $\textit{RATD}$
method is more straightforward to apply and produces the strongest results in
the unseen setting on which we focus. Our single best Reasoning model using
only 440 million parameters materially improves upon strong comparable prior
baselines for unseen evaluation datasets (StrategyQA 58.9 $\rightarrow$ 61.7
acc., CommonsenseQA 63.6 $\rightarrow$ 72.7 acc., ARC-DA 31.6 $\rightarrow$
52.1 F1, IIRC 25.5 $\rightarrow$ 27.3 F1) and a version utilising our prior
knowledge of each type of question in selecting a context combination strategy
does even better. Our proposed models also generally outperform direct prompts
against much larger models (BLOOM 175B and StableVicuna 13B) in both few-shot
chain-of-thought and few-shot answer-only settings.

在提供足够解释性上下文的情况下，较小的语言模型在挑战性的问答任务中表现出强大的推理能力，而问题在训练中是未见过的。我们评估了两种进一步改进这一设置的方法，具体为使用理由排名模型对生成的合理和真实合理性进行评分并使用得分来从知识源中生成组合上下文，以及通过训练较小的推理模型来利用长文本序列中的相关信息。普遍发现，两种方法都有效，但 RATD 方法更容易应用并在我们关注的未见设置中产生最佳结果。