Modern NLP tasks increasingly rely on dense retrieval methods to access
up-to-date and relevant contextual information. We are motivated by the premise
that retrieval benefits from segments that can vary in size such that a
content's semantic independence is better captured. We propose LumberChunker, a
method leveraging an LLM to dynamically segment documents, which iteratively
prompts the LLM to identify the point within a group of sequential passages
where the content begins to shift. To evaluate our method, we introduce
GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer
pairs derived from 100 public domain narrative books available on Project
Gutenberg. Our experiments show that LumberChunker not only outperforms the
most competitive baseline by 7.37% in retrieval performance (DCG@20) but also
that, when integrated into a RAG pipeline, LumberChunker proves to be more
effective than other chunking methods and competitive baselines, such as the
Gemini 1.5M Pro. Our Code and Data are available at
this https URL

利用 LumberChunker 方法切割文档，以动态地对文档进行分割，通过将其整合到 RAG 管道中，证明了在检索性能和效果上优于其他切割方法和竞争基准的竞争基准。