The study of multimodal interaction in therapy can yield a comprehensive understanding of therapist and patient behavior that can be used to develop a multimodal virtual agent supporting therapy. This investigation aims to uncover how therapists skillfully blend therapy's task goal (employing classical steps of Motivational Interviewing) with the social goal (building a trusting relationship and expressing empathy). Furthermore, we seek to categorize patients into various ``types'' requiring tailored therapeutic approaches. To this intent, we present multimodal annotations of a corpus consisting of simulated motivational interviewing conversations, wherein actors portray the roles of patients and therapists. We introduce EMMI, composed of two publicly available MI corpora, AnnoMI and the Motivational Interviewing Dataset, for which we add multimodal annotations. We analyze these annotations to characterize functional behavior for developing a virtual agent performing motivational interviews emphasizing social and empathic behaviors. Our analysis found three clusters of patients expressing significant differences in behavior and adaptation of the therapist's behavior to those types. This shows the importance of a therapist being able to adapt their behavior depending on the current situation within the dialog and the type of user.

通过研究多模式交互在治疗中的应用，可以深入理解治疗师和患者的行为，从而开发支持治疗的多模式虚拟代理。本研究旨在揭示治疗师如何巧妙地将治疗任务目标（应用经典的激励性面谈步骤）与社交目标（建立信任关系和表达同理心）相结合，并将患者分为不同的“类型”，需要量身定制的治疗方法。通过多模式注释的模拟激励性面谈对话语料库，本研究分析了治疗师和患者的功能行为，以开发强调社交和同理心行为的多模式虚拟代理。分析结果发现了三类在行为和治疗师对这些类型的行为和适应性的差异方面表现出显著差异的患者群体，展示了治疗师根据对话的当前情况和用户类型能够调整自身行为的重要性。

EMMI -- 恻隐的多模态动机性访谈数据集：分析与标注