We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions about Pittsburgh and Carnegie Mellon University (CMU). We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs, achieving an inter-annotator agreement (IAA) score of 0.7625. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy. Experimental results show that the RAG system significantly outperforms a non-RAG baseline, particularly in time-sensitive and complex queries, with an F1 score improvement from 5.45% to 42.21% and recall of 56.18%. This study demonstrates the potential of RAG systems in enhancing answer precision and relevance, while identifying areas for further optimization in document retrieval and model training.

本研究解决了领域特定问答中信息检索的不足，通过设计检索增强生成（RAG）系统，为大型语言模型提供有关匹兹堡和卡内基梅隆大学的相关文档。研究结果表明，该系统在复杂和时间敏感的查询中显著提升了答案的准确性和相关性，F1评分从5.45%提升至42.21%。

基于检索增强生成的领域特定问答：以匹兹堡和卡内基梅隆大学为案例研究