Large Language Models (LLM) have shown remarkable language capabilities
fueling attempts to integrate them into applications across a wide range of
domains. An important application area is question answering over private
enterprise documents where the main considerations are data security, which
necessitates applications that can be deployed on-prem, limited computational
resources and the need for a robust application that correctly responds to
queries. Retrieval-Augmented Generation (RAG) has emerged as the most prominent
framework for building LLM-based applications. While building a RAG is
relatively straightforward, making it robust and a reliable application
requires extensive customization and relatively deep knowledge of the
application domain. We share our experiences building and deploying an LLM
application for question answering over private organizational documents. Our
application combines the use of RAG with a finetuned open-source LLM.
Additionally, our system, which we call Tree-RAG (T-RAG), uses a tree structure
to represent entity hierarchies within the organization. This is used to
generate a textual description to augment the context when responding to user
queries pertaining to entities within the organization's hierarchy. Our
evaluations show that this combination performs better than a simple RAG or
finetuning implementation. Finally, we share some lessons learned based on our
experiences building an LLM application for real-world use.

构建一个基于 RAG 的 LLM 应用程序，Tree-RAG 使用树结构表示组织中的实体层级，并生成文字描述以增强响应用户查询的上下文。