Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.

该研究提出了一个多模态数据集，用于个人经历分享中的共情研究，其中包含41名参与者在与人工智能代理共享脆弱经历并阅读共情共鸣故事期间的53小时视频、音频和文本数据。该数据集是首个关于共情的纵向数据集，通过社交机器人在参与者家中一个月的使用中收集，在自然的共情叙事互动中参与者与AI代理进行交互。研究引入一项新的任务，基于个人经历预测个体对他人故事的共情，评估其在个人共享故事环境和他们阅读故事的反思中的效果。研究为未来在上下文化和纵向共情建模方面的改进铺平了道路，并为发展富有共情能力的AI系统和理解真实世界情境下人类共情的复杂性提供了宝贵资源。

EmpathicStories++: 一个面向个人经历的多模态数据集