We propose a novel perspective of viewing large pretrained models as search
engines, thereby enabling the repurposing of techniques previously used to
enhance search engine performance. As an illustration, we employ a personalized
query rewriting technique in the realm of text-to-image generation. Despite
significant progress in the field, it is still challenging to create
personalized visual representations that align closely with the desires and
preferences of individual users. This process requires users to articulate
their ideas in words that are both comprehensible to the models and accurately
capture their vision, posing difficulties for many users. In this paper, we
tackle this challenge by leveraging historical user interactions with the
system to enhance user prompts. We propose a novel approach that involves
rewriting user prompts based a new large-scale text-to-image dataset with over
300k prompts from 3115 users. Our rewriting model enhances the expressiveness
and alignment of user prompts with their intended visual outputs. Experimental
results demonstrate the superiority of our methods over baseline approaches, as
evidenced in our new offline evaluation method and online tests. Our approach
opens up exciting possibilities of applying more search engine techniques to
build truly personalized large pretrained models.

我们提出了一种将大型预训练模型视为搜索引擎的新视角，从而使以前用于提高搜索引擎性能的技术得以重新应用。通过个性化查询重写技术的示例，我们在文本到图像生成领域使用了这种视角。尽管该领域取得了显著进展，但仍然很难创建与个体用户的期望和偏好密切对齐的个性化视觉表示。这个过程需要用户用对模型理解且准确捕捉他们的愿景的文字来表达，这对许多用户来说很困难。在本文中，我们通过利用用户与系统的历史互动来提高用户提示的质量来解决这个挑战。我们提出了一种基于具有超过 300k 个提示的 3115 个用户的新大规模文本到图像数据集的用户提示重写方法。我们的重写模型增强了用户提示与其预期视觉输出之间的表达和对齐。实验证明了我们方法的优越性，这在我们的新离线评估方法和在线测试中得到了证明。我们的方法为应用更多搜索引擎技术来构建真正个性化的大型预训练模型打开了令人兴奋的可能性。

个性化的改写提示增强文本生成图像

Tailored Visions: Enhancing Text-to-Image Generation with Personalized  Prompt Rewriting

Conversational AI systems (e.g. Alexa, Siri, Google Assistant, etc.) need to
understand queries with defects to ensure robust conversational understanding
and reduce user frictions. The defective queries are often induced by user
ambiguities and mistakes, or errors in the automatic speech recognition (ASR)
and natural language understanding (NLU).
Personalized query rewriting (personalized QR) targets reducing defects in
the torso and tail user query traffic, and it typically relies on an index of
past successful user interactions with the conversational AI. This paper
presents our "Collaborative Query Rewriting" approach that focuses on rewriting
novel user interactions unseen in the user history. This approach builds a
"user Feedback Interaction Graph" (FIG) consisting of historical user-entity
interactions, and leverages multi-hop customer affinity to enrich each user's
index (i.e. the Collaborative User Index) that would help cover future unseen
defective queries. To counteract the precision degradation from the enlarged
index, we introduced additional transformer layers to the L1 retrieval model
and added multi-hop affinity and guardrail features to the L2 re-ranking model.
Given the production constraints of storage cost and runtime retrieval
latency, managing the size of the Collaborative User Index is important. As the
user index can be pre-computed, we explored using a Large Language Model (LLM)
for multi-hop customer affinity retrieval on the Video/Music domains. In
particular, this paper looked into the Dolly-V2 7B model. Given limited user
index size, We found the user index derived from fine-tuned Dolly-V2 generation
significantly enhanced coverage of unseen user interactions. Consequently, this
boosted QR performance on unseen user interactions compared to the graph
traversal based user index.

本文介绍了一种名为 Collaborative Query Rewriting 的个性化查询重写方法，该方法利用过去用户与对话 AI 的交互来增强用户索引，采用了多跳关系亲近度和警戒特征等技术，通过使用大型语言模型来提高多跳关系亲近度检索。通过实验证明，该方法显著提高了对未知用户交互的 QR 性能。