Learned sparse retrieval (LSR) is a family of neural methods that encode
queries and documents into sparse lexical vectors that can be indexed and
retrieved efficiently with an inverted index. We explore the application of LSR
to the multi-modal domain, with a focus on text-image retrieval. While LSR has
seen success in text retrieval, its application in multimodal retrieval remains
underexplored. Current approaches like LexLIP and STAIR require complex
multi-step training on massive datasets. Our proposed approach efficiently
transforms dense vectors from a frozen dense model into sparse lexical vectors.
We address issues of high dimension co-activation and semantic deviation
through a new training algorithm, using Bernoulli random variables to control
query expansion. Experiments with two dense models (BLIP, ALBEF) and two
datasets (MSCOCO, Flickr30k) show that our proposed algorithm effectively
reduces co-activation and semantic deviation. Our best-performing sparsified
model outperforms state-of-the-art text-image LSR models with a shorter
training time and lower GPU memory requirements. Our approach offers an
effective solution for training LSR retrieval models in multimodal settings.
Our code and model checkpoints are available at
github.com/thongnt99/lsr-multimodal

通过利用 Bernoulli 随机变量控制查询扩展，我们提出的训练算法能够有效减少高维共同激活和语义偏差，并在多模态设置中训练了一种有效的 LSR (learned sparse retrieval) 检索模型，该模型在训练时间和 GPU 内存需求方面优于现有的文本 - 图像 LSR 模型。

多模态学习稀疏检索与概率扩展控制

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

Learned sparse retrieval (LSR) is a family of neural retrieval methods that
transform queries and documents into sparse weight vectors aligned with a
vocabulary. While LSR approaches like Splade work well for short passages, it
is unclear how well they handle longer documents. We investigate existing
aggregation approaches for adapting LSR to longer documents and find that
proximal scoring is crucial for LSR to handle long documents. To leverage this
property, we proposed two adaptations of the Sequential Dependence Model (SDM)
to LSR: ExactSDM and SoftSDM. ExactSDM assumes only exact query term
dependence, while SoftSDM uses potential functions that model the dependence of
query terms and their expansion terms (i.e., terms identified using a
transformer's masked language modeling head).
Experiments on the MSMARCO Document and TREC Robust04 datasets demonstrate
that both ExactSDM and SoftSDM outperform existing LSR aggregation approaches
for different document length constraints. Surprisingly, SoftSDM does not
provide any performance benefits over ExactSDM. This suggests that soft
proximity matching is not necessary for modeling term dependence in LSR.
Overall, this study provides insights into handling long documents with LSR,
proposing adaptations that improve its performance.

本研究探讨了通过使用近似评分适应长文档的已有聚合方法、并给出了将顺序依赖模型（SDM）应用于学习到的稀疏检索（LSR）的两种改进方法：ExactSDM 和 SoftSDM。结果表明，这些改进方法优于现有的 LSR 聚合方法，并且 SoftSDM 对 ExactSDM 并没有提供任何性能优势。这表明，软接近匹配在 LSR 中不是建模词语依赖性的必要条件。