In mixed-initiative conversational search systems, clarifying questions are
used to help users who struggle to express their intentions in a single query.
These questions aim to uncover user's information needs and resolve query
ambiguities. We hypothesize that in scenarios where multimodal information is
pertinent, the clarification process can be improved by using non-textual
information. Therefore, we propose to add images to clarifying questions and
formulate the novel task of asking multimodal clarifying questions in
open-domain, mixed-initiative conversational search systems. To facilitate
research into this task, we collect a dataset named Melon that contains over 4k
multimodal clarifying questions, enriched with over 14k images. We also propose
a multimodal query clarification model named Marto and adopt a prompt-based,
generative fine-tuning strategy to perform the training of different stages
with different prompts. Several analyses are conducted to understand the
importance of multimodal contents during the query clarification phase.
Experimental results indicate that the addition of images leads to significant
improvements of up to 90% in retrieval performance when selecting the relevant
images. Extensive analyses are also performed to show the superiority of Marto
compared with discriminative baselines in terms of effectiveness and
efficiency.

在混合主动交互搜索系统中，通过添加图像到澄清问题并提出多模态澄清问题的新任务，可以显著提高查询澄清过程的效果，改善信息检索性能，从而有效地解决了用户在表达意图和解决查询模糊性方面的困难。

多模态澄清问题在混合主动对话式搜索中的应用

Asking Multimodal Clarifying Questions in Mixed-Initiative  Conversational Search

While the body of research directed towards constructing and generating
clarifying questions in mixed-initiative conversational search systems is vast,
research aimed at processing and comprehending users' answers to such questions
is scarce. To this end, we present a simple yet effective method for processing
answers to clarifying questions, moving away from previous work that simply
appends answers to the original query and thus potentially degrades retrieval
performance. Specifically, we propose a classifier for assessing usefulness of
the prompted clarifying question and an answer given by the user. Useful
questions or answers are further appended to the conversation history and
passed to a transformer-based query rewriting module. Results demonstrate
significant improvements over strong non-mixed-initiative baselines.
Furthermore, the proposed approach mitigates the performance drops when non
useful questions and answers are utilized.

我们提出了一个简单而有效的方法来处理对澄清问题的回答，并介绍了一个对用户提出的问题和回答进行有用性评估的分类器，将有用的问题或回答附加到对话历史中，并传递给基于 transformer 的查询重写模块，实验证明与非混合主动的基线相比有显著改进。此外，所提出的方法缓解了使用不实用的问题和回答时性能下降的问题。