The introduction of Vec2Text, a technique for inverting text embeddings, has
raised serious privacy concerns within dense retrieval systems utilizing text
embeddings, including those provided by OpenAI and Cohere. This threat comes
from the ability for a malicious attacker with access to text embeddings to
reconstruct the original text.
In this paper, we investigate various aspects of embedding models that could
influence the recoverability of text using Vec2Text. Our exploration involves
factors such as distance metrics, pooling functions, bottleneck pre-training,
training with noise addition, embedding quantization, and embedding dimensions
-- aspects not previously addressed in the original Vec2Text paper. Through a
thorough analysis of these factors, our aim is to gain a deeper understanding
of the critical elements impacting the trade-offs between text recoverability
and retrieval effectiveness in dense retrieval systems. This analysis provides
valuable insights for practitioners involved in designing privacy-aware dense
retrieval systems. Additionally, we propose a straightforward fix for embedding
transformation that ensures equal ranking effectiveness while mitigating the
risk of text recoverability.
Furthermore, we extend the application of Vec2Text to the separate task of
corpus poisoning, where, theoretically, Vec2Text presents a more potent threat
compared to previous attack methods. Notably, Vec2Text does not require access
to the dense retriever's model parameters and can efficiently generate numerous
adversarial passages.
In summary, this study highlights the potential threat posed by Vec2Text to
existing dense retrieval systems, while also presenting effective methods to
patch and strengthen such systems against such risks.

该研究论文探讨了 Vec2Text 的潜在威胁和强化方法，关注于密集检索系统中的嵌入模型、隐私问题和转换技术。

理解和缓解 Vec2Text 对密集检索系统的威胁

Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval  Systems

Masked auto-encoder pre-training has emerged as a prevalent technique for
initializing and enhancing dense retrieval systems. It generally utilizes
additional Transformer decoder blocks to provide sustainable supervision
signals and compress contextual information into dense representations.
However, the underlying reasons for the effectiveness of such a pre-training
technique remain unclear. The usage of additional Transformer-based decoders
also incurs significant computational costs. In this study, we aim to shed
light on this issue by revealing that masked auto-encoder (MAE) pre-training
with enhanced decoding significantly improves the term coverage of input tokens
in dense representations, compared to vanilla BERT checkpoints. Building upon
this observation, we propose a modification to the traditional MAE by replacing
the decoder of a masked auto-encoder with a completely simplified Bag-of-Word
prediction task. This modification enables the efficient compression of lexical
signals into dense representations through unsupervised pre-training.
Remarkably, our proposed method achieves state-of-the-art retrieval performance
on several large-scale retrieval benchmarks without requiring any additional
parameters, which provides a 67% training speed-up compared to standard masked
auto-encoder pre-training with enhanced decoding.

通过使用增强解码的遮蔽自动编码器预训练，显著提高了稠密表示中输入标记的术语覆盖，从而实现了在大规模检索基准上的最先进的检索性能，无需任何额外参数，相比于使用增强解码的标准遮蔽自动编码器预训练速度提高了 67%。

放弃您的解码器：使用词袋预测进行密集段落检索的预训练

Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense  Passage Retrieval

The integration of external personalized context information into
document-grounded conversational systems has significant potential business
value, but has not been well-studied. Motivated by the concept of personalized
context-aware document-grounded conversational systems, we introduce the task
of context-aware passage retrieval. We also construct a dataset specifically
curated for this purpose. We describe multiple baseline systems to address this
task, and propose a novel approach, Personalized Context-Aware Search (PCAS),
that effectively harnesses contextual information during passage retrieval.
Experimental evaluations conducted on multiple popular dense retrieval systems
demonstrate that our proposed approach not only outperforms the baselines in
retrieving the most relevant passage but also excels at identifying the
pertinent context among all the available contexts. We envision that our
contributions will serve as a catalyst for inspiring future research endeavors
in this promising direction.

个性化上下文感知的文档导向对话系统的集成具有重要的商业价值，但尚未得到充分研究。通过个性化上下文感知的文档导向对话系统的概念，我们引入了上下文感知的段落检索任务，并构建了一个专门为此目的策划的数据集。我们描述了多个基线系统来解决这个任务，并提出了一种新颖的方法，个性化上下文感知搜索（PCAS），在段落检索过程中有效利用上下文信息。在多个常见的密集检索系统上进行的实证评估表明，我们提出的方法不仅在检索相关段落方面优于基线系统，而且在识别所有可用上下文中的相关环境方面表现出色。我们预计我们的工作将成为激发未来研究在这个有前景的方向上的催化剂。