Retrieval-augmented generation (RAG) is a promising way to improve large
language models (LLMs) for generating more factual, accurate, and up-to-date
content. Existing methods either optimize prompts to guide LLMs in leveraging
retrieved information or directly fine-tune the LLMs to adapt to RAG scenarios.
Although fine-tuning can yield better performance, it often compromises the
LLMs' general generation capabilities by modifying their parameters. This
limitation poses challenges in practical applications, especially when LLMs are
already deployed, as parameter adjustments may affect their original
functionality. To address this, we propose a novel method that involves
learning scalable and pluggable virtual tokens for RAG. By maintaining the
LLMs' original parameters and fine-tuning only the embeddings of these
pluggable tokens, our approach not only enhances LLMs' performance but also
preserves their general generation capacities. Furthermore, we design several
training strategies to improve the scalability, flexibility, and
generalizability of our method. Comprehensive experiments across nine
question-answering tasks demonstrate the superiority of our approach.

通过学习可扩展且可插拔的虚拟标记，我们的方法在保持大型语言模型的原始参数的基础上，仅对这些可插入标记的嵌入进行微调，从而提高了大型语言模型的性能并保留了其普遍的生成能力。

一个令牌可以帮助！学习可扩展和可插拔的虚拟令牌用于增强检索的大型语言模型

One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for  Retrieval-Augmented Large Language Models

Text style transfer is a prominent task that aims to control the style of
text without inherently changing its factual content. To cover more text
modification applications, such as adapting past news for current events and
repurposing educational materials, we propose the task of text fact transfer,
which seeks to transfer the factual content of a source text between topics
without modifying its style. We find that existing language models struggle
with text fact transfer, due to their inability to preserve the specificity and
phrasing of the source text, and tendency to hallucinate errors. To address
these issues, we design ModQGA, a framework that minimally modifies a source
text with a novel combination of end-to-end question generation and
specificity-aware question answering. Through experiments on four existing
datasets adapted for text fact transfer, we show that ModQGA can accurately
transfer factual content without sacrificing the style of the source text.

通过最小化修改源文本的问答生成方式，ModQGA 能够准确转移到事实内容，同时不丢失源文本的风格。

文本事实传递

Text Fact Transfer

This work aims to build a dialogue agent that can weave new factual content
into conversations as naturally as humans. We draw insights from linguistic
principles of conversational analysis and annotate human-human conversations
from the Switchboard Dialog Act Corpus to examine humans strategies for
acknowledgement, transition, detail selection and presentation. When current
chatbots (explicitly provided with new factual content) introduce facts into a
conversation, their generated responses do not acknowledge the prior turns.
This is because models trained with two contexts - new factual content and
conversational history - generate responses that are non-specific w.r.t. one of
the contexts, typically the conversational history. We show that specificity
w.r.t. conversational history is better captured by Pointwise Conditional
Mutual Information ($\text{pcmi}_h$) than by the established use of Pointwise
Mutual Information ($\text{pmi}$). Our proposed method, Fused-PCMI, trades off
$\text{pmi}$ for $\text{pcmi}_h$ and is preferred by humans for overall quality
over the Max-PMI baseline 60% of the time. Human evaluators also judge
responses with higher $\text{pcmi}_h$ better at acknowledgement 74% of the
time. The results demonstrate that systems mimicking human conversational
traits (in this case acknowledgement) improve overall quality and more broadly
illustrate the utility of linguistic principles in improving dialogue agents.

本论文旨在构建一种对话代理，能够像人一样自然地将新的事实内容编织到对话中。我们从对话分析的语言学原理中吸取灵感，并对 Switchboard 对话行为语料库中的人类对话进行注释，以检查人类关于承认、转移、细节选择和展示的策略。我们提出的方法，混合 PCMI，通过交换 pmi 和 pcmi_h 来取得更好的性能，被人类评估员 60％的时间内更喜欢。