The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.

通过引入一个包含2965个真实世界文档和29590个专家注释的问答对的基准测试套件，我们重新审视了解析数据和检索的设计选择以及多个文档领域和不同查询类型中答案质量的流行的LLM和RAG解决方案，我们的评估得出了有趣的发现，并强调了数据解析和检索的重要性。

UDA：一套用于现实世界文档分析中检索增强生成的基准测试套件