We fine-tune GPT-3 to answer long-form questions using a text-based
web-browsing environment, which allows the model to search and navigate the
web. By setting up the task so that it can be performed by humans, we are able
to train models on the task using imitation learning, and then optimize answer
quality with human feedback. To make human evaluation of factual accuracy
easier, models must collect references while browsing in support of their
answers. We train and evaluate our models on ELI5, a dataset of questions asked
by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior
cloning, and then performing rejection sampling against a reward model trained
to predict human preferences. This model's answers are preferred by humans 56%
of the time to those of our human demonstrators, and 69% of the time to the
highest-voted answer from Reddit.

在文本浏览环境中使用 GPT-3 进行长篇问题回答的模型训练，通过模仿学习和人类反馈优化答案质量，并使用 ELI5 数据集对模型进行评估和训练，最佳模型通过行为克隆和逆向采样获得，最终得到的答案比 Reddit 上评分最高答案及人类演示者的答案更受人类用户喜欢。

WebGPT: 借助浏览器进行问题回答并获得人类反馈

WebGPT: Browser-assisted question-answering with human feedback

We introduce the first large-scale corpus for long-form question answering, a
task requiring elaborate and in-depth answers to open-ended questions. The
dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five''
(ELI5) where an online community provides answers to questions which are
comprehensible by five year olds. Compared to existing datasets, ELI5 comprises
diverse questions requiring multi-sentence answers. We provide a large set of
web documents to help answer the question. Automatic and human evaluations show
that an abstractive model trained with a multi-task objective outperforms
conventional Seq2Seq, language modeling, as well as a strong extractive
baseline. However, our best model is still far from human performance since
raters prefer gold responses in over 86% of cases, leaving ample opportunity
for future improvement.

这篇论文介绍了第一个大规模的长篇问答语料库，其中包括了 270,000 个 Reddit 论坛上的线索，使用多任务目标的诠释性模型表现最佳，但仍存在改进的余地。