Evaluating retrieval-augmented generation (RAG) systems traditionally relies
on hand annotations for input queries, passages to retrieve, and responses to
generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating
RAG systems along the dimensions of context relevance, answer faithfulness, and
answer relevance. Using synthetic training data, ARES finetunes lightweight LM
judges to assess the quality of individual RAG components. To mitigate
potential prediction errors, ARES utilizes a small set of human-annotated
datapoints for prediction-powered inference (PPI). Across six different
knowledge-intensive tasks in KILT and SuperGLUE, ARES accurately evaluates RAG
systems while using a few hundred human annotations during evaluation.
Furthermore, ARES judges remain effective across domain shifts, proving
accurate even after changing the type of queries and/or documents used in the
evaluated RAG systems. We make our datasets and code for replication and
deployment available at this https URL

通过使用综合训练数据，ARES 通过微调轻量级语言模型评估 RAG 组件的质量，在 KILT 和 SuperGLUE 两个领域多任务中准确评估 RAG 系统的有效性。

ARES：一种用于检索增强生成系统的自动化评估框架

ARES: An Automated Evaluation Framework for Retrieval-Augmented  Generation Systems

Topic drift is a common phenomenon in multi-turn dialogue. Therefore, an
ideal dialogue generation models should be able to capture the topic
information of each context, detect the relevant context, and produce
appropriate responses accordingly. However, existing models usually use word or
sentence level similarities to detect the relevant contexts, which fail to well
capture the topical level relevance. In this paper, we propose a new model,
named STAR-BTM, to tackle this problem. Firstly, the Biterm Topic Model is
pre-trained on the whole training dataset. Then, the topic level attention
weights are computed based on the topic representation of each context.
Finally, the attention weights and the topic distribution are utilized in the
decoding process to generate the corresponding responses. Experimental results
on both Chinese customer services data and English Ubuntu dialogue data show
that STAR-BTM significantly outperforms several state-of-the-art methods, in
terms of both metric-based and human evaluations.

本文提出了一种基于 Biterm Topic Model（BTM）的对话生成模型 STAR-BTM 来解决多轮对话中话题漂移的问题，并在中文客服数据和英文 Ubuntu 对话数据上进行了实验，结果表明 STAR-BTM 在度量和人工评估方面明显优于现有模型。