To address the challenges of scalable and intelligent question-answering (QA), we introduce an innovative solution that leverages open-source Large Language Models (LLMs) to ensure data privacy. We use models from the LLaMA-2 family and augmentations including retrieval augmented generation (RAG), supervised fine-tuning (SFT), and an alternative to reinforcement learning with human feedback (RLHF). We perform our experiments on a Piazza dataset from an introductory CS course with 10k QA pairs and 1.5k pairs of preferences data and conduct both human evaluations and automatic LLM evaluations on a small subset. We find preliminary evidence that modeling techniques collectively enhance the quality of answers by 33%, and RAG is an impactful addition. This work paves the way for the development of ChaTA, an intelligent QA assistant customizable for courses with an online QA platform.

为应对可扩展且智能的问答挑战，我们引入了一种创新解决方案，利用开源的大型语言模型（LLMs）来确保数据隐私。我们在一个入门计算机科学课程的 Piazza 数据集上进行了实验，并对一个小的子集进行了人工评估和自动 LLM 评估。我们初步发现通过多种建模技术能够将答案的质量提升33％，而 RAG 是一个有影响力的改进。这项工作为开发适用于在线问答平台的智能 QA 助手 ChaTA 打下了基础。

ChaTA：运用开源LLMs构建智能问答辅导助手