In this paper, we introduce the semantic knowledge of medical images from their diagnostic reports to provide an inspirational network training and an interpretable prediction mechanism with our proposed novel multimodal neural network, namely TandemNet. Inside TandemNet, a language model is used to represent report text, which cooperates with the image model in a tandem scheme. We propose a novel dual-attention model that facilitates high-level interactions between visual and semantic information and effectively distills useful features for prediction. In the testing stage, TandemNet can make accurate image prediction with an optional report text input. It also interprets its prediction by producing attention on the image and text informative feature pieces, and further generating diagnostic report paragraphs. Based on a pathological bladder cancer images and their diagnostic reports (BCIDR) dataset, sufficient experiments demonstrate that our method effectively learns and integrates knowledge from multimodalities and obtains significantly improved performance than comparing baselines.

该论文介绍了一种名为TandemNet的新型多模态神经网络，利用医学影像的语义知识和诊断报告进行训练，提出了一种新的双注意力模型以实现视觉和语义信息之间的高层交互，并在测试阶段准确地预测图像，并通过产生诊断报告自动解释其预测。实验结果表明，该方法从多模态中学习和集成知识，并获得比基准方法更好的性能。

TandemNet：使用诊断报告作为可选语义参考从医学影像中提取知识