We present a comprehensive evaluation of answer quality in
Retrieval-Augmented Generation (RAG) applications using vRAG-Eval, a novel
grading system that is designed to assess correctness, completeness, and
honesty. We further map the grading of quality aspects aforementioned into a
binary score, indicating an accept or reject decision, mirroring the intuitive
"thumbs-up" or "thumbs-down" gesture commonly used in chat applications. This
approach suits factual business settings where a clear decision opinion is
essential. Our assessment applies vRAG-Eval to two Large Language Models
(LLMs), evaluating the quality of answers generated by a vanilla RAG
application. We compare these evaluations with human expert judgments and find
a substantial alignment between GPT-4's assessments and those of human experts,
reaching 83% agreement on accept or reject decisions. This study highlights the
potential of LLMs as reliable evaluators in closed-domain, closed-ended
settings, particularly when human evaluations require significant resources.

我们提出了一种全面评估检索增强生成（RAG）应用中答案质量的方法，使用 vRAG-Eval，这是一种新的评分系统，旨在评估正确性、完整性和诚实性。我们进一步将前述质量方面的评分转化为一个二进制分数，表示接受或拒绝的决策，反映了常用于聊天应用的直观 “赞” 或 “踩” 的手势。我们将 vRAG-Eval 应用于两个大型语言模型（LLM），评估由基本 RAG 应用生成的答案的质量。我们将这些评估与人类专家判断进行比较，并发现 GPT-4 的评估结果与人类专家的评判具有显著一致性，在接受或拒绝的决策上达成 83% 的一致。这项研究突出了 LLM 在封闭领域、封闭式问题设置中作为可靠评估者的潜力，特别是当人工评估需要大量资源时。

评估检索增强生成模型的答案质量：强大的 LLM 足够了

Evaluating Quality of Answers for Retrieval-Augmented Generation: A  Strong LLM Is All You Need

In this thesis, we investigated the relevance, faithfulness, and succinctness
aspects of Long Form Question Answering (LFQA). LFQA aims to generate an
in-depth, paragraph-length answer for a given question, to help bridge the gap
between real scenarios and the existing open-domain QA models which can only
extract short-span answers. LFQA is quite challenging and under-explored. Few
works have been done to build an effective LFQA system. It is even more
challenging to generate a good-quality long-form answer relevant to the query
and faithful to facts, since a considerable amount of redundant, complementary,
or contradictory information will be contained in the retrieved documents.
Moreover, no prior work has been investigated to generate succinct answers. We
are among the first to research the LFQA task. We pioneered the research
direction to improve the answer quality in terms of 1) query-relevance, 2)
answer faithfulness, and 3) answer succinctness.

本论文研究了长篇问答系统的相关性、可靠性和简洁性，旨在改进问题相关性、答案可靠性和简洁性等方面的答案质量。

生成式长篇问答：相关性、忠实度和简洁性

Generative Long-form Question Answering: Relevance, Faithfulness and Succinctness

Product-specific community question answering platforms can greatly help
address the concerns of potential customers. However, the user-provided answers
on such platforms often vary a lot in their qualities. Helpfulness votes from
the community can indicate the overall quality of the answer, but they are
often missing. Accurately predicting the helpfulness of an answer to a given
question and thus identifying helpful answers is becoming a demanding need.
Since the helpfulness of an answer depends on multiple perspectives instead of
only topical relevance investigated in typical QA tasks, common answer
selection algorithms are insufficient for tackling this task. In this paper, we
propose the Review-guided Answer Helpfulness Prediction (RAHP) model that not
only considers the interactions between QA pairs but also investigates the
opinion coherence between the answer and crowds' opinions reflected in the
reviews, which is another important factor to identify helpful answers.
Moreover, we tackle the task of determining opinion coherence as a language
inference problem and explore the utilization of pre-training strategy to
transfer the textual inference knowledge obtained from a specifically designed
trained network. Extensive experiments conducted on real-world data across
seven product categories show that our proposed model achieves superior
performance on the prediction task.

本文提出了一种基于评论引导的答案有用性预测模型，该模型不仅考虑了 QA 对之间的交互，还探讨了答案与众筹意见之间的意见连贯性作为识别有用答案的另一个重要因素，并探索了利用预训练策略将从一个特定设计的经过训练的网络中获得的文本推理知识转移的方法。该模型在七个产品类别上进行的广泛实验表明，其在预测任务中的表现优异。