Computational methods for depression detection aim to mine traces of
depression from online publications posted by Internet users. However,
solutions trained on existing collections exhibit limited generalisation and
interpretability. To tackle these issues, recent studies have shown that
identifying depressive symptoms can lead to more robust models. The eRisk
initiative fosters research on this area and has recently proposed a new
ranking task focused on developing search methods to find sentences related to
depressive symptoms. This search challenge relies on the symptoms specified by
the Beck Depression Inventory-II (BDI-II), a questionnaire widely used in
clinical practice. Based on the participant systems' results, we present the
DepreSym dataset, consisting of 21580 sentences annotated according to their
relevance to the 21 BDI-II symptoms. The labelled sentences come from a pool of
diverse ranking methods, and the final dataset serves as a valuable resource
for advancing the development of models that incorporate depressive markers
such as clinical symptoms. Due to the complex nature of this relevance
annotation, we designed a robust assessment methodology carried out by three
expert assessors (including an expert psychologist). Additionally, we explore
here the feasibility of employing recent Large Language Models (ChatGPT and
GPT4) as potential assessors in this complex task. We undertake a comprehensive
examination of their performance, determine their main limitations and analyze
their role as a complement or replacement for human annotators.

通过挖掘互联网用户发布的在线出版物中的抑郁症迹象，计算机方法可以进行抑郁症检测，但是现有集合上训练的解决方案具有较有限的概括性和可解释性。为解决这些问题，最近的研究表明，通过识别抑郁症状可以得到更稳健的模型。eRisk 倡议在这一领域推动研究，并最近提出了一个新的排名任务，重点是开发搜索方法来找到与抑郁症状相关的句子。这个搜索挑战依赖于贝克抑郁症量表（BDI-II）指定的症状，该量表被广泛应用于临床实践。根据参与系统的结果，我们推出了 DepreSym 数据集，该数据集包含 21580 个句子，根据其与 21 个 BDI-II 症状的相关性进行了注释。这些标记的句子来自各种排名方法的池，最终数据集是推进模型开发的宝贵资源，其中包括临床症状等抑郁标记。由于相关性注释的复杂性质，我们设计了一个由三名专家评估员（包括一位专业心理学家）进行的强大评估方法。此外，我们在这里探讨了使用最近的大型语言模型（ChatGPT 和 GPT4）作为潜在评估员的可行性，对其性能进行了全面的考察，确定了它们的主要局限性，并分析了它们作为人类标注者的补充或替代角色。