Image search stands as a pivotal task in multimedia and computer vision,
finding applications across diverse domains, ranging from internet search to
medical diagnostics. Conventional image search systems operate by accepting
textual or visual queries, retrieving the top-relevant candidate results from
the database. However, prevalent methods often rely on single-turn procedures,
introducing potential inaccuracies and limited recall. These methods also face
the challenges, such as vocabulary mismatch and the semantic gap, constraining
their overall effectiveness. To address these issues, we propose an interactive
image retrieval system capable of refining queries based on user relevance
feedback in a multi-turn setting. This system incorporates a vision language
model (VLM) based image captioner to enhance the quality of text-based queries,
resulting in more informative queries with each iteration. Moreover, we
introduce a large language model (LLM) based denoiser to refine text-based
query expansions, mitigating inaccuracies in image descriptions generated by
captioning models. To evaluate our system, we curate a new dataset by adapting
the MSR-VTT video retrieval dataset to the image retrieval task, offering
multiple relevant ground truth images for each query. Through comprehensive
experiments, we validate the effectiveness of our proposed system against
baseline methods, achieving state-of-the-art performance with a notable 10\%
improvement in terms of recall. Our contributions encompass the development of
an innovative interactive image retrieval system, the integration of an
LLM-based denoiser, the curation of a meticulously designed evaluation dataset,
and thorough experimental validation.

我们提出了一种互动式图像检索系统，结合了视觉语言模型和大型语言模型，通过用户反馈迭代改进查询，并利用无噪声的查询扩展提高检索准确性，在评估中获得了 10% 的召回率改善。

利用大型语言模型和视觉语言模型增强交互式图像检索的查询重写

Enhancing Interactive Image Retrieval With Query Rewriting Using Large  Language Models and Vision Language Models

Automatic speech recognition (ASR) systems can suffer from poor recall for
various reasons, such as noisy audio, lack of sufficient training data, etc.
Previous work has shown that recall can be improved by retrieving rewrite
candidates from a large database of likely, contextually-relevant alternatives
to the hypothesis text using nearest-neighbors search over embeddings of the
ASR hypothesis text to correct and candidate corrections.
However, ASR-hypothesis-based retrieval can yield poor precision if the
textual hypotheses are too phonetically dissimilar to the transcript truth. In
this paper, we eliminate the hypothesis-audio mismatch problem by querying the
correction database directly using embeddings derived from the utterance audio;
the embeddings of the utterance audio and candidate corrections are produced by
multimodal speech-text embedding networks trained to place the embedding of the
audio of an utterance and the embedding of its corresponding textual transcript
close together.
After locating an appropriate correction candidate using nearest-neighbor
search, we score the candidate with its speech-text embedding distance before
adding the candidate to the original n-best list.
We show a relative word error rate (WER) reduction of 6% on utterances whose
transcripts appear in the candidate set, without increasing WER on general
utterances.

通过使用多模式语音文本嵌入网络，从语音文本数据库中直接查询校正候选项，以消除音频 - 假设不匹配问题，并在将候选项添加到原始候选列表之前使用语音 - 文本嵌入距离对候选项进行评分，该方法在改善检索率的同时，降低了相对的词错误率（WER）。