Multimodal search has become increasingly important in providing users with a
natural and effective way to ex-press their search intentions. Images offer
fine-grained details of the desired products, while text allows for easily
incorporating search modifications. However, some existing multimodal search
systems are unreliable and fail to address simple queries. The problem becomes
harder with the large variability of natural language text queries, which may
contain ambiguous, implicit, and irrelevant in-formation. Addressing these
issues may require systems with enhanced matching capabilities, reasoning
abilities, and context-aware query parsing and rewriting. This paper introduces
a novel multimodal search model that achieves a new performance milestone on
the Fashion200K dataset. Additionally, we propose a novel search interface
integrating Large Language Models (LLMs) to facilitate natural language
interaction. This interface routes queries to search systems while
conversationally engaging with users and considering previous searches. When
coupled with our multimodal search model, it heralds a new era of shopping
assistants capable of offering human-like interaction and enhancing the overall
search experience.

多模态搜索系统结合图像和文本，通过增强匹配能力、推理能力和上下文感知的查询解析和重写，提供用户与其搜索意图自然有效的交互。在 Fashion200K 数据集上，我们引入了一种新颖的多模态搜索模型，并提出了结合大型语言模型的搜索界面，以实现与用户的对话式互动和上下文考虑的搜索体验的升级，为购物助手提供了类人交互和全面的搜索体验。

大规模语言模型在多模态搜索中的应用

Leveraging Large Language Models for Multimodal Search

This paper reviews the state-of-the-art of hybrid language models
architectures and strategies for "complex" question-answering (QA, CQA, CPS).
Large Language Models (LLM) are good at leveraging public data on standard
problems but once you want to tackle more specific complex questions or
problems you may need specific architecture, knowledge, skills, methods,
sensitive data protection, explainability, human approval and versatile
feedback... We identify key elements augmenting LLM to solve complex questions
or problems. We extend findings from the robust community edited research
papers BIG, BLOOM and HELM which open source, benchmark and analyze limits and
challenges of LLM in terms of tasks complexity and strict evaluation on
accuracy (e.g. fairness, robustness, toxicity, ...). Recent projects like
ChatGPT and GALACTICA have allowed non-specialists to grasp the great potential
as well as the equally strong limitations of language models in complex QA.
Hybridizing these models with different components could allow to overcome
these different limits and go much further. We discuss some challenges
associated with complex QA, including domain adaptation, decomposition and
efficient multi-step QA, long form and non-factoid QA, safety and
multi-sensitivity data protection, multimodal search, hallucinations,
explainability and truthfulness, temproal reasoning. Therefore, we analyze
current solutions and promising research trends, using elements such as: hybrid
LLM architectures, active human reinforcement learning supervised with AI,
prompting adaptation, neuro-symbolic and structured knowledge grounding,
program synthesis, iterated decomposition and others.

本文回顾了混合语言模型在复杂问答（QA，CQA，CPS）中架构和策略的最新进展，指出了提高 LLM 在处理背景知识、安全数据保护、解释性等方面的方法，并探讨了与复杂 QA 相关的挑战及当前解决方案和发展趋势。