How humans can efficiently and effectively acquire images has always been a
perennial question. A typical solution is text-to-image retrieval from an
existing database given the text query; however, the limited database typically
lacks creativity. By contrast, recent breakthroughs in text-to-image generation
have made it possible to produce fancy and diverse visual content, but it faces
challenges in synthesizing knowledge-intensive images. In this work, we rethink
the relationship between text-to-image generation and retrieval and propose a
unified framework in the context of Multimodal Large Language Models (MLLMs).
Specifically, we first explore the intrinsic discriminative abilities of MLLMs
and introduce a generative retrieval method to perform retrieval in a
training-free manner. Subsequently, we unify generation and retrieval in an
autoregressive generation way and propose an autonomous decision module to
choose the best-matched one between generated and retrieved images as the
response to the text query. Additionally, we construct a benchmark called
TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
the evaluation of unified text-to-image generation and retrieval. Extensive
experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
proposed method.

研究中，我们提出了一个统一框架，以多模态大型语言模型（MLLMs）为背景，探索了文本到图像生成和检索之间的关系，并引入了一种生成检索方法，在无需训练的情况下进行检索。我们还构建了一个基准测试集 TIGeR-Bench 以标准化统一的文本到图像生成和检索方法的评估，并在 TIGeR-Bench 以及两个检索基准测试集 Flickr30K 和 MS-COCO 上进行了大量实验，证明了我们提出方法的优越性和效果。

统一的文本到图像生成和检索

Unified Text-to-Image Generation and Retrieval

In this paper, we present a generative retrieval method for sponsored search
engine, which uses neural machine translation (NMT) to generate keywords
directly from query. This method is completely end-to-end, which skips query
rewriting and relevance judging phases in traditional retrieval systems.
Different from standard machine translation, the target space in the retrieval
setting is a constrained closed set, where only committed keywords should be
generated. We present a Trie-based pruning technique in beam search to address
this problem. The biggest challenge in deploying this method into a real
industrial environment is the latency impact of running the decoder.
Self-normalized training coupled with Trie-based dynamic pruning dramatically
reduces the inference time, yielding a speedup of more than 20 times. We also
devise an mixed online-offline serving architecture to reduce the latency and
CPU consumption. To encourage the NMT to generate new keywords uncovered by the
existing system, training data is carefully selected. This model has been
successfully applied in Baidu's commercial search engine as a supplementary
retrieval branch, which has brought a remarkable revenue improvement of more
than 10 percents.

本篇论文提出了一种使用神经机器翻译方法来直接从查询中生成关键字的检索方法，并采用基于 Trie 的修剪技术来解决目标空间为限制闭合集合的检索环境下的关键字生成问题。该方法被应用于百度商业搜索引擎作为辅助检索分支，使得收益提升 10% 以上。