The Query-By-Document (QBD) problem is an information retrieval problem where the query is a document, and the retrieved candidates are documents that match the query document, often in a domain or query specific manner. This can be crucial for tasks such as patent matching, legal or compliance case retrieval, and academic literature review. Existing retrieval methods, including keyword search and document embeddings, can be optimized with domain-specific datasets to improve QBD search performance. However, creating these domain-specific datasets is often costly and time-consuming. Our work introduces a process to generate custom QBD-search datasets and compares a set of methods to use in this problem, which we refer to as QBD-RankedDatagen. We provide a comparative analysis of our proposed methods in terms of cost, speed, and the human interface with the domain experts. The methods we compare leverage Large Language Models (LLMs) which can incorporate domain expert input to produce document scores and rankings, as well as explanations for human review. The process and methods for it that we present can significantly reduce human effort in dataset creation for custom domains while still obtaining sufficient expert knowledge for tuning retrieval models. We evaluate our methods on QBD datasets from the Text Retrieval Conference (TREC) and finetune the parameters of the BM25 model -- which is used in many industrial-strength search engines like OpenSearch -- using the generated data.

本研究解决了生成特定领域的基于文档查询（QBD）数据集所需的高成本和时间消耗问题。我们提出了一种新的QBD排名数据生成方法，利用大型语言模型（LLM）结合领域专家的输入来生成文档评分和排名，从而显著减少了数据集创建中的人力投入，同时仍能获取足够的专家知识。研究表明，该方法能够提高QBD搜索的性能并优化检索模型的参数。 

QBD排名数据生成：利用LLM重排序以减少人工成本生成定制排名数据集以改善基于文档查询的搜索