While deep networks have achieved broad success in analyzing natural images, when applied to medical scans, they often fail in unexcepted situations. We investigate this challenge and focus on model sensitivity to domain shifts, such as data sampled from different hospitals or data confounded by demographic variables such as sex, race, etc, in the context of chest X-rays and skin lesion images. A key finding we show empirically is that existing visual backbones lack an appropriate prior from the architecture for reliable generalization in these settings. Taking inspiration from medical training, we propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language. To this end, we introduce Knowledge-enhanced Bottlenecks (KnoBo), a class of concept bottleneck models that incorporates knowledge priors that constrain it to reason with clinically relevant factors found in medical textbooks or PubMed. KnoBo uses retrieval-augmented language models to design an appropriate concept space paired with an automatic training procedure for recognizing the concept. We evaluate different resources of knowledge and recognition architectures on a broad range of domain shifts across 20 datasets. In our comprehensive evaluation with two imaging modalities, KnoBo outperforms fine-tuned models on confounded datasets by 32.4% on average. Finally, evaluations reveal that PubMed is a promising resource for making medical models less sensitive to domain shift, outperforming other resources on both diversity of information and final prediction performance.

医学图像数据中，现有深度网络模型对于不同医院采样的数据或受性别、种族等人口统计变量影响的数据领域转移问题的鲁棒性不足。本研究提出了一种基于医学知识先验的知识增强瓶颈模型（KnoBo），通过检索增强语言模型，设计适当的概念空间，并采用自动训练过程来识别概念，以提高医学模型对领域转移的适应能力。在广泛的20个数据集上评估了不同知识资源和识别架构，在混淆数据集上，KnoBo的性能平均超过精调模型32.4%。最后，评估表明PubMed是使医学模型对领域转移的敏感性降低的有希望的资源，在信息多样性和最终预测性能上优于其他资源。

领域偏移的教科书解决方案：医学图像分析的知识先验