Agents represent one of the most emerging applications of Large Language
Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal
capabilities to navigate complex user environments. Conversational Health
Agents (CHAs), a prime example of this, are redefining healthcare by offering
nuanced support that transcends textual analysis to incorporate emotional
intelligence. This paper introduces an LLM-based CHA engineered for rich,
multimodal dialogue-especially in the realm of mental health support. It
adeptly interprets and responds to users' emotional states by analyzing
multimodal cues, thus delivering contextually aware and empathetically resonant
verbal responses. Our implementation leverages the versatile openCHA framework,
and our comprehensive evaluation involves neutral prompts expressed in diverse
emotional tones: sadness, anger, and joy. We evaluate the consistency and
repeatability of the planning capability of the proposed CHA. Furthermore,
human evaluators critique the CHA's empathic delivery, with findings revealing
a striking concordance between the CHA's outputs and evaluators' assessments.
These results affirm the indispensable role of vocal (soon multimodal) emotion
recognition in strengthening the empathetic connection built by CHAs, cementing
their place at the forefront of interactive, compassionate digital health
solutions.

本文介绍了一种基于大型语言模型和生成式人工智能的会话式健康代理，通过分析多模态线索，它可以解释和回应用户的情绪状态，从而提供具有情境意识和共鸣力的交流回应，从而巩固交互式、富有同情心的数字健康解决方案的前沿地位。

对话界面中的多模态共情

Empathy Through Multimodality in Conversational Interfaces

Conversational Health Agents (CHAs) are interactive systems designed to
enhance personal healthcare services by engaging in empathetic conversations
and processing multimodal data. While current CHAs, especially those utilizing
Large Language Models (LLMs), primarily focus on conversation, they often lack
comprehensive agent capabilities. This includes the ability to access personal
user health data from wearables, 24/7 data collection sources, and electronic
health records, as well as integrating the latest published health insights and
connecting with established multimodal data analysis tools. We are developing a
framework to empower CHAs by equipping them with critical thinking, knowledge
acquisition, and problem-solving abilities. Our CHA platform, powered by LLMs,
seamlessly integrates healthcare tools, enables multilingual and multimodal
conversations, and interfaces with a variety of user data analysis tools. We
illustrate its proficiency in handling complex healthcare tasks, such as stress
level estimation, showcasing the agent's cognitive and operational
capabilities.

我们正在开发一个框架，通过赋予对话健康代理（CHAs）关键思维、知识获取和问题解决能力，以增强个人医疗保健服务，并展示其在处理复杂医疗任务中的熟练性和操作能力。