We introduce the Variational Open-Domain (VOD) framework for end-to-end
training and evaluation of retrieval-augmented models (open-domain question
answering and language modelling). We show that the Rényi variational bound,
a lower bound to the task marginal likelihood, can be exploited to aid
optimization and use importance sampling to estimate the task log-likelihood
lower bound and its gradients using samples drawn from an auxiliary retriever
(approximate posterior). The framework can be used to train modern
retrieval-augmented systems end-to-end using tractable and consistent estimates
of the Rényi variational bound and its gradients. We demonstrate the
framework's versatility by training reader-retriever BERT-based models on
multiple-choice medical exam questions (MedMCQA and USMLE). We registered a new
state-of-the-art for both datasets (MedMCQA: $62.9$\%, USMLE: $55.0$\%). Last,
we show that the retriever part of the learned reader-retriever model trained
on the medical board exam questions can be used in search engines for a medical
knowledge base.

本文介绍了 Variational Open-Domain（VOD）框架，用于训练和评估检索增强模型（开放领域问答和语言建模），研究了 Rényi 变分界限，使用重要性采样从辅助检索器抽取样本，并利用该框架在多项选择医学考试问题上训练了读者 - 检索器 BERT 模型。

变分开放领域问答

Variational Open-Domain Question Answering

Although large language models (LLMs) often produce impressive outputs, it
remains unclear how they perform in real-world scenarios requiring strong
reasoning skills and expert domain knowledge. We set out to investigate whether
GPT-3.5 (Codex and InstructGPT) can be applied to answer and reason about
difficult real-world-based questions. We utilize two multiple-choice medical
exam questions (USMLE and MedMCQA) and a medical reading comprehension dataset
(PubMedQA). We investigate multiple prompting scenarios: Chain-of-Thought (CoT,
think step-by-step), zero- and few-shot (prepending the question with
question-answer exemplars) and retrieval augmentation (injecting Wikipedia
passages into the prompt). For a subset of the USMLE questions, a medical
expert reviewed and annotated the model's CoT. We found that InstructGPT can
often read, reason and recall expert knowledge. Failure are primarily due to
lack of knowledge and reasoning errors and trivial guessing heuristics are
observed, e.g.\ too often predicting labels A and D on USMLE. Sampling and
combining many completions overcome some of these limitations. Using 100
samples, Codex 5-shot CoT not only gives close to well-calibrated predictive
probability but also achieves human-level performances on the three datasets.
USMLE: 60.2%, MedMCQA: 62.7% and PubMedQA: 78.2%.

研究了 GPT-3.5 在实际世界中具有强大推理能力和专业领域知识的应用能力，使用了多项提示方法（CoT、零和少量训练以及检索增强），并在医学考试和阅读理解领域取得了人类水平表现。