Efficient knowledge management plays a pivotal role in augmenting both the
operational efficiency and the innovative capacity of businesses and
organizations. By indexing knowledge through vectorization, a variety of
knowledge retrieval methods have emerged, significantly enhancing the efficacy
of knowledge management systems. Recently, the rapid advancements in generative
natural language processing technologies paved the way for generating precise
and coherent answers after retrieving relevant documents tailored to user
queries. However, for enterprise knowledge bases, assembling extensive training
data from scratch for knowledge retrieval and generation is a formidable
challenge due to the privacy and security policies of private data, frequently
entailing substantial costs. To address the challenge above, in this paper, we
propose EKRG, a novel Retrieval-Generation framework based on large language
models (LLMs), expertly designed to enable question-answering for Enterprise
Knowledge bases with limited annotation costs. Specifically, for the retrieval
process, we first introduce an instruction-tuning method using an LLM to
generate sufficient document-question pairs for training a knowledge retriever.
This method, through carefully designed instructions, efficiently generates
diverse questions for enterprise knowledge bases, encompassing both
fact-oriented and solution-oriented knowledge. Additionally, we develop a
relevance-aware teacher-student learning strategy to further enhance the
efficiency of the training process. For the generation process, we propose a
novel chain of thought (CoT) based fine-tuning method to empower the LLM-based
generator to adeptly respond to user questions using retrieved documents.
Finally, extensive experiments on real-world datasets have demonstrated the
effectiveness of our proposed framework.

通过使用大规模语言模型（LLMs），我们提出了一种名为 EKRG 的新型检索 - 生成框架，以极低的注释成本实现了企业知识库的问答。通过独特的指令调优方法和针对连贯性思维的微调方法，我们的框架在真实世界数据集上进行了广泛实验，并取得了显著的有效性。