In many situations, we have both rich- and poor- data environments: in a rich-data environment (e.g., intensive care units), we have high-quality multi-modality data. On the other hand, in a poor-data environment (e.g., at home), we often only have access to a single data modality with low quality. How can we learn an accurate and efficient model for the poor-data environment by leveraging multi-modality data from the rich-data environment? In this work, we propose a knowledge distillation model RDPD to enhance a small model trained on poor data with a complex model trained on rich data. In an end-to-end fashion, RDPD trains a student model built on a single modality data (poor data) to imitate the behavior and performance of a teacher model from multimodal data (rich data) via jointly optimizing the combined loss of attention imitation and target imitation. We evaluated RDPD on three real-world datasets. RDPD consistently outperformed all baselines across all three datasets, especially achieving the greatest performance improvement over a standard neural network model trained on the common features (Direct model) by 24.56% on PR-AUC and 12.21% on ROC-AUC, and over the standard knowledge distillation model by 5.91% on PR-AUC and 4.44% on ROC-AUC.

该研究开发和提出了一种知识蒸馏（KD）方法（RDPD）来利用从丰富的私有数据训练的高复杂模型的蒸馏知识来增强在贫数据环境中训练的预测模型，在三个真实数据集上评估 RDPD ，并证明其蒸馏模型在所有数据集上一致优于所有基线模型，特别是在 PR-AUC 上比仅使用低质量数据训练的模型提高 24.56％，在ROC-AUC上提高了12.21％，并且比现有技术的 KD 模型优化了5.91％的 PR-AUC 和 4.44％的 ROC-AUC.

RDPD: 通过模仿使用富数据支持贫数据