With the emergence of large language models (LLMs), investigating if they can
surpass humans in areas such as emotion recognition and empathetic responding
has become a focal point of research. This paper presents a comprehensive study
exploring the empathetic responding capabilities of four state-of-the-art LLMs:
GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, and Mixtral-8x7B-Instruct in
comparison to a human baseline. We engaged 1,000 participants in a
between-subjects user study, assessing the empathetic quality of responses
generated by humans and the four LLMs to 2,000 emotional dialogue prompts
meticulously selected to cover a broad spectrum of 32 distinct positive and
negative emotions. Our findings reveal a statistically significant superiority
of the empathetic responding capability of LLMs over humans. GPT-4 emerged as
the most empathetic, marking approximately 31% increase in responses rated as
"Good" compared to the human benchmark. It was followed by LLaMA-2,
Mixtral-8x7B, and Gemini-Pro, which showed increases of approximately 24%, 21%,
and 10% in "Good" ratings, respectively. We further analyzed the response
ratings at a finer granularity and discovered that some LLMs are significantly
better at responding to specific emotions compared to others. The suggested
evaluation framework offers a scalable and adaptable approach for assessing the
empathy of new LLMs, avoiding the need to replicate this study's findings in
future research.

本研究通过与人类基准对比的方式，综合评估了四个最先进的大型语言模型（GPT-4、LLaMA-2、Mixtral-8x7B 和 Gemini-Pro）在共 2000 个情感对话提示中的回应中的移情能力。我们的发现显示，大型语言模型在移情回应能力上显著超过人类，其中 GPT-4 的回应被评为 “好” 的比例比人类基准提高了约 31%。此外，我们发现不同的大型语言模型在回应不同情感时表现显著不同。基于研究结果，我们提出了一种可扩展和可适应的评估框架，用于评估新大型语言模型的移情能力，避免了未来研究中重复这项研究的需求。