Figure drawing is often used as part of dementia screening protocols. The
Survey of Health Aging and Retirement in Europe (SHARE) has adopted three
drawing tests from Addenbrooke's Cognitive Examination III as part of its
questionnaire module on cognition. While the drawings are usually scored by
trained clinicians, SHARE uses the face-to-face interviewers who conduct the
interviews to score the drawings during fieldwork. This may pose a risk to data
quality, as interviewers may be less consistent in their scoring and more
likely to make errors due to their lack of clinical training. This paper
therefore reports a first proof of concept and evaluates the feasibility of
automating scoring using deep learning. We train several different
convolutional neural network (CNN) models using about 2,000 drawings from the
8th wave of the SHARE panel in Germany and the corresponding interviewer
scores, as well as self-developed 'gold standard' scores. The results suggest
that this approach is indeed feasible. Compared to training on interviewer
scores, models trained on the gold standard data improve prediction accuracy by
about 10 percentage points. The best performing model, ConvNeXt Base, achieves
an accuracy of about 85%, which is 5 percentage points higher than the accuracy
of the interviewers. While this is a promising result, the models still
struggle to score partially correct drawings, which are also problematic for
interviewers. This suggests that more and better training data is needed to
achieve production-level prediction accuracy. We therefore discuss possible
next steps to improve the quality and quantity of training examples.

使用深度学习自动评分来提高数据质量，训练适应痴呆筛查的模型需要更多和更好的训练数据。

认知绘图的自动评分：基于机器评分与黄金标准的质量评估

Automatic Scoring of Cognition Drawings: Assessing the Quality of  Machine-Based Scores Against a Gold Standard

For dementia screening and monitoring, standardized tests play a key role in
clinical routine since they aim at minimizing subjectivity by measuring
performance on a variety of cognitive tasks. In this paper, we report on a
study that consists of a semi-standardized history taking followed by two
standardized neuropsychological tests, namely the SKT and the CERAD-NB. The
tests include basic tasks such as naming objects, learning word lists, but also
widely used tools such as the MMSE. Most of the tasks are performed verbally
and should thus be suitable for automated scoring based on transcripts. For the
first batch of 30 patients, we analyze the correlation between expert manual
evaluations and automatic evaluations based on manual and automatic
transcriptions. For both SKT and CERAD-NB, we observe high to perfect
correlations using manual transcripts; for certain tasks with lower
correlation, the automatic scoring is stricter than the human reference since
it is limited to the audio. Using automatic transcriptions, correlations drop
as expected and are related to recognition accuracy; however, we still observe
high correlations of up to 0.98 (SKT) and 0.85 (CERAD-NB). We show that using
word alternatives helps to mitigate recognition errors and subsequently
improves correlation with expert scores.

该研究通过对 30 名患者的语音转录，对比专家手动评估和自动评估之间的相关性，探讨了半标准化病史采集后进行 SKT 和 CERAD-NB 两个标准化认知测试的效果；结果表明，使用自动化转录可以得到较高的相关性，而使用词汇替换技术可以缓解识别误差，进一步提高与专家评分的相关性。