Multimodal search has become increasingly important in providing users with a
natural and effective way to ex-press their search intentions. Images offer
fine-grained details of the desired products, while text allows for easily
incorporating search modifications. However, some existing multimodal search
systems are unreliable and fail to address simple queries. The problem becomes
harder with the large variability of natural language text queries, which may
contain ambiguous, implicit, and irrelevant in-formation. Addressing these
issues may require systems with enhanced matching capabilities, reasoning
abilities, and context-aware query parsing and rewriting. This paper introduces
a novel multimodal search model that achieves a new performance milestone on
the Fashion200K dataset. Additionally, we propose a novel search interface
integrating Large Language Models (LLMs) to facilitate natural language
interaction. This interface routes queries to search systems while
conversationally engaging with users and considering previous searches. When
coupled with our multimodal search model, it heralds a new era of shopping
assistants capable of offering human-like interaction and enhancing the overall
search experience.

多模态搜索系统结合图像和文本，通过增强匹配能力、推理能力和上下文感知的查询解析和重写，提供用户与其搜索意图自然有效的交互。在 Fashion200K 数据集上，我们引入了一种新颖的多模态搜索模型，并提出了结合大型语言模型的搜索界面，以实现与用户的对话式互动和上下文考虑的搜索体验的升级，为购物助手提供了类人交互和全面的搜索体验。