This paper addresses the quality of annotations in mental health datasets
used for NLP-based depression level estimation from social media texts. While
previous research relies on social media-based datasets annotated with binary
categories, i.e. depressed or non-depressed, recent datasets such as D2S and
PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of
these datasets rely on crowd workers without the domain knowledge for
annotation. Focusing on the PRIMATE dataset, our study reveals concerns
regarding annotation validity, particularly for the lack of interest or
pleasure symptom. Through reannotation by a mental health professional, we
introduce finer labels and textual spans as evidence, identifying a notable
number of false positives. Our refined annotations, to be released under a Data
Use Agreement, offer a higher-quality test set for anhedonia detection. This
study underscores the necessity of addressing annotation quality issues in
mental health datasets, advocating for improved methodologies to enhance NLP
model reliability in mental health assessments.

本研究关注于精神健康数据集中的注释质量，特别是使用社交媒体文本进行基于自然语言处理的抑郁水平估计。通过重新注释和引入更精细的标签和文本范围，我们发现针对 PRIMATE 数据集的注释有效性存在关注点，特别是缺乏兴趣或愉悦症状，并识别出了相当数量的假阳性。我们通过发布经过精细标注的数据集，在应用数据使用协议的前提下，为无情感体验检测提供了更高质量的测试集。本研究强调了在精神健康数据集中解决注释质量问题的必要性，并倡导改进方法论以提高自然语言处理模型在精神健康评估中的可靠性。