Amidst the growing interest in developing task-autonomous AI for automated
mental health care, this paper addresses the ethical and practical challenges
associated with the issue and proposes a structured framework that delineates
levels of autonomy, outlines ethical requirements, and defines beneficial
default behaviors for AI agents in the context of mental health support. We
also evaluate ten state-of-the-art language models using 16 mental
health-related questions designed to reflect various mental health conditions,
such as psychosis, mania, depression, suicidal thoughts, and homicidal
tendencies. The question design and response evaluations were conducted by
mental health clinicians (M.D.s). We find that existing language models are
insufficient to match the standard provided by human professionals who can
navigate nuances and appreciate context. This is due to a range of issues,
including overly cautious or sycophantic responses and the absence of necessary
safeguards. Alarmingly, we find that most of the tested models could cause harm
if accessed in mental health emergencies, failing to protect users and
potentially exacerbating existing symptoms. We explore solutions to enhance the
safety of current models. Before the release of increasingly task-autonomous AI
systems in mental health, it is crucial to ensure that these models can
reliably detect and manage symptoms of common psychiatric disorders to prevent
harm to users. This involves aligning with the ethical framework and default
behaviors outlined in our study. We contend that model developers are
responsible for refining their systems per these guidelines to safeguard
against the risks posed by current AI technologies to user mental health and
safety.
Trigger warning: Contains and discusses examples of sensitive mental health
topics, including suicide and self-harm.

通过对有关心理健康支持的批判性问题的评估，本文提出了一个结构化框架，旨在应对与自主任务 AI 在心理健康领域的发展所相关的伦理和实际挑战，并定义了 AI 代理在支持心理健康方面的等级自主性、伦理要求和有益的默认行为。此外，评估了十种先进的语言模型，结果发现这些现有模型不能达到人类专业人员的标准，不能适应细微差别和理解上下文，可能导致用户受伤甚至加重症状。因此，需要探索解决方案来增强当前模型的安全性，以确保它们能可靠地检测和管理常见心理障碍的症状，从而避免对用户的伤害。这需要与我们研究中提出的伦理框架和默认行为相一致，强调模型开发人员有责任根据这些指南改进其系统，以防止当前人工智能技术对用户心理健康和安全造成风险。