This study introduces De-DSI, a novel framework that fuses large language
models (LLMs) with genuine decentralization for information retrieval,
particularly employing the differentiable search index (DSI) concept in a
decentralized setting. Focused on efficiently connecting novel user queries
with document identifiers without direct document access, De-DSI operates
solely on query-docid pairs. To enhance scalability, an ensemble of DSI models
is introduced, where the dataset is partitioned into smaller shards for
individual model training. This approach not only maintains accuracy by
reducing the number of data each model needs to handle but also facilitates
scalability by aggregating outcomes from multiple models. This aggregation uses
a beam search to identify top docids and applies a softmax function for score
normalization, selecting documents with the highest scores for retrieval. The
decentralized implementation demonstrates that retrieval success is comparable
to centralized methods, with the added benefit of the possibility of
distributing computational complexity across the network. This setup also
allows for the retrieval of multimedia items through magnet links, eliminating
the need for platforms or intermediaries.

De-DSI 是一种新颖的框架，通过将大型语言模型（LLMs）与信息检索真正的分布式化相结合，特别是在分布式环境中采用可微分搜索索引（DSI）的概念，以有效地将新颖的用户查询与文档标识符连接起来，在仅使用查询 - 文档标识符对上进行操作。该框架通过引入 DSI 模型的集合来提高可扩展性，其中数据集被分割为较小的片段进行单独的模型训练。通过使用波束搜索来识别顶级文档标识符并应用 softmax 函数进行评分归一化，从而选择具有最高得分的文档进行检索。该分布式实现表明检索成功率与集中式方法相当，同时具有将计算复杂性分布到网络中的额外好处。该设置还允许通过磁力链接检索多媒体项目，消除了平台或中介的需求。