High-quality psychological counseling is crucial for mental health worldwide,
and timely evaluation is vital for ensuring its effectiveness. However,
obtaining professional evaluation for each counseling session is expensive and
challenging. Existing methods that rely on self or third-party manual reports
to assess the quality of counseling suffer from subjective biases and
limitations of time-consuming.
To address above challenges, this paper proposes an innovative and efficient
automatic approach using large language models (LLMs) to evaluate the working
alliance in counseling conversations. We collected a comprehensive counseling
dataset and conducted multiple third-party evaluations based on therapeutic
relationship theory. Our LLM-based evaluation, combined with our guidelines,
shows high agreement with human evaluations and provides valuable insights into
counseling scripts. This highlights the potential of LLMs as supervisory tools
for psychotherapists. By integrating LLMs into the evaluation process, our
approach offers a cost-effective and dependable means of assessing counseling
quality, enhancing overall effectiveness.

提出使用大型语言模型进行心理咨询工作联结评估的创新高效自动方法，通过结合指南，该方法在与人工评估的高度一致性的基础上提供有价值的心理咨询脚本洞察，将大型语言模型整合到评估过程中，提供一种经济实惠可靠的评估咨询质量的手段，增强整体效果。

利用 LLMs 进行心理健康咨询的自动评估

Automatic Evaluation for Mental Health Counseling using LLMs

As a predictive measure of the treatment outcome in psychotherapy, the
working alliance measures the agreement of the patient and the therapist in
terms of their bond, task and goal. Long been a clinical quantity estimated by
the patients' and therapists' self-evaluative reports, we believe that the
working alliance can be better characterized using natural language processing
technique directly in the dialogue transcribed in each therapy session. In this
work, we propose the Working Alliance Transformer (WAT), a Transformer-based
classification model that has a psychological state encoder which infers the
working alliance scores by projecting the embedding of the dialogues turns onto
the embedding space of the clinical inventory for working alliance. We evaluate
our method in a real-world dataset with over 950 therapy sessions with anxiety,
depression, schizophrenia and suicidal patients and demonstrate an empirical
advantage of using information about the therapeutic states in this sequence
classification task of psychotherapy dialogues.

本文提出了一种基于 Transformer 的工作联盟分类模型，其具有心理状态编码器，通过将对话转换成嵌入到工作联盟的临床清单的空间中，推断出工作联盟得分，实现了通过自然语言处理技术直接表征工作联盟。在一个涉及多种疾病的真实世界数据集上进行的实验结果表明，使用治疗状态信息，可以在心理治疗对话的序列分类任务中获得实质性的优势。