We present a framework - Prompt, Generate, Train (PGT) - to efficiently develop a generative question-answering model for open-book question-answering over a proprietary collection of text documents. The framework adapts a retriever augmented generation model to the target domain using supervised finetuning and reinforcement learning with synthetic feedback in a few-shot setting. This yields an aligned, uncertainty calibrated model that is competitive with GPT-4 based in-context retrieval augmented generation in generating relevant answers at lower serving costs. The synthetic generation pipeline generates high quality synthetic training data musing a medium sized LLM, Flan-T5 XXL, and a novel consistency filtering scheme. The pipeline is designed to generate both abstractive and extractive questions that span the entire corpus. Using samples from this dataset, the framework fine-tunes a smaller RAG model comprising a dense retriever and a smaller sized LLM on samples from the dataset. In parallel, the framework trains a Reward model to score domain grounded answers higher than hallucinated answers. In the next phase, the framework aligns to the RAG model with the target domain using reinforcement learning. This step improves the RAG model's ability to generate grounded answers and ignore out of domain questions. In the final phase, the framework calibrates the model uncertainty for extractive question-answers. This is a desirable feature since the model can be integrated into a cascading system where the RAG model's answer is surfaced only when the model is confident of its answer.

本研究提出了Prompt，Generate，Train（PGT）框架，用于高效开发基于收集的文本文档的生成式问答模型，并利用监督微调和强化学习方法构建出一个与GPT-4类似的生成式QA模型。

Prompt Generate Train (PGT): A framework for few-shot domain adaptation,
  alignment, and uncertainty calibration of a retriever augmented generation
  (RAG) model for domain specific open book question-answering

Prompt Generate Train（PGT）: 一种少样本领域自适应、对齐和不确定性校准框架，用于增强式生成器（RAG）模型在特定领域开放式书籍问答中的应用