The USA Food and Drug Administration has accorded increasing importance to patient-reported problems in clinical and research settings. In this paper, we explore one of the largest online datasets comprising 170,141 open-ended self-reported responses (called "verbatims") from patients with Parkinson's (PwPs) to questions about what bothers them about their Parkinson's Disease and how it affects their daily functioning, also known as the Parkinson's Disease Patient Report of Problems. Classifying such verbatims into multiple clinically relevant symptom categories is an important problem and requires multiple steps - expert curation, a multi-label text classification (MLTC) approach and large amounts of labelled training data. Further, human annotation of such large datasets is tedious and expensive. We present a novel solution to this problem where we build a baseline dataset using 2,341 (of the 170,141) verbatims annotated by nine curators including clinical experts and PwPs. We develop a rules based linguistic-dictionary using NLP techniques and graph database-based expert phrase-query system to scale the annotation to the remaining cohort generating the machine annotated dataset, and finally build a Keras-Tensorflow based MLTC model for both datasets. The machine annotated model significantly outperforms the baseline model with a F1-score of 95% across 65 symptom categories on a held-out test set.

本文探讨了由美国FDA所重视的临床和研究背景中患者报告的问题，结合一个包含17万个PwPs（Parkinson's with Patients）在线开放式问答中的人工日志，从中分类临床相关的症状，并使用一个基于规则的语言字典和NLP技术的专家短语查询系统来缩放注释，最后构建了一个基于Keras-TensorFlow的多标签文本分类模型，通过在测试集上的F1-score达到了95％以上。

What Do Patients Say About Their Disease Symptoms? Deep Multilabel Text
  Classification With Human-in-the-Loop Curation for Automatic Labeling of
  Patient Self Reports of Problems

患者对其疾病症状的描述是什么？利用人工辅助标记进行深度多标签文本分类，自动标记患者自述问题