Retrieval-Augmented Generation (RAG) offers a promising solution to address
various limitations of Large Language Models (LLMs), such as hallucination and
difficulties in keeping up with real-time updates. This approach is
particularly critical in expert and domain-specific applications where LLMs
struggle to cover expert knowledge. Therefore, evaluating RAG models in such
scenarios is crucial, yet current studies often rely on general knowledge
sources like Wikipedia to assess the models' abilities in solving common-sense
problems. In this paper, we evaluated LLMs by RAG settings in a domain-specific
context, college enrollment. We identified six required abilities for RAG
models, including the ability in conversational RAG, analyzing structural
information, faithfulness to external knowledge, denoising, solving
time-sensitive problems, and understanding multi-document interactions. Each
ability has an associated dataset with shared corpora to evaluate the RAG
models' performance. We evaluated popular LLMs such as Llama, Baichuan,
ChatGLM, and GPT models. Experimental results indicate that existing
closed-book LLMs struggle with domain-specific questions, highlighting the need
for RAG models to solve expert problems. Moreover, there is room for RAG models
to improve their abilities in comprehending conversational history, analyzing
structural information, denoising, processing multi-document interactions, and
faithfulness in expert knowledge. We expect future studies could solve these
problems better.

检验 Retrieval-Augmented Generation （RAG）在特定领域环境中的能力与效果，评估存在的限制并寻求改进的研究。

DomainRAG: 用于评估领域特定的检索增强生成的中文基准

DomainRAG: A Chinese Benchmark for Evaluating Domain-specific  Retrieval-Augmented Generation

The emerging trend of advancing generalist artificial intelligence, such as
GPTv4 and Gemini, has reshaped the landscape of research (academia and
industry) in machine learning and many other research areas. However,
domain-specific applications of such foundation models (e.g., in medicine)
remain untouched or often at their very early stages. It will require an
individual set of transfer learning and model adaptation techniques by further
expanding and injecting these models with domain knowledge and data. The
development of such technologies could be largely accelerated if the bundle of
data, algorithms, and pre-trained foundation models were gathered together and
open-sourced in an organized manner. In this work, we present OpenMEDLab, an
open-source platform for multi-modality foundation models. It encapsulates not
only solutions of pioneering attempts in prompting and fine-tuning large
language and vision models for frontline clinical and bioinformatic
applications but also building domain-specific foundation models with
large-scale multi-modal medical data. Importantly, it opens access to a group
of pre-trained foundation models for various medical image modalities, clinical
text, protein engineering, etc. Inspiring and competitive results are also
demonstrated for each collected approach and model in a variety of benchmarks
for downstream tasks. We welcome researchers in the field of medical artificial
intelligence to continuously contribute cutting-edge methods and models to
OpenMEDLab, which can be accessed via this https URL

OpenMEDLab 是一个开放源代码平台，用于多模态基础模型，它不仅集成了前沿临床和生物信息学应用的大型语言和视觉模型的解决方案，还利用大规模多模态医学数据构建领域特定的基础模型。该平台还提供了各种医学图像模态、临床文本和蛋白工程等领域的预训练基础模型，并在各种基准测试中展示了激动人心的竞争结果。欢迎医学人工智能领域的研究人员不断为 OpenMEDLab 贡献前沿的方法和模型。