Deep learning based Computer Aided Diagnosis (CAD) systems have been developed to treat breast ultrasound. Most of them focus on a single ultrasound imaging modality, either using representative static images or the dynamic video of a real-time scan. In fact, these two image modalities are complementary for lesion diagnosis. Dynamic videos provide detailed three-dimensional information about the lesion, while static images capture the typical sections of the lesion. In this work, we propose a multi-modality breast tumor diagnosis model to imitate the diagnosing process of radiologists, which learns the features of both static images and dynamic video and explores the potential relationship between the two modalities. Considering that static images are carefully selected by professional radiologists, we propose to aggregate dynamic video features under the guidance of domain knowledge from static images before fusing multi-modality features. Our work is validated on a breast ultrasound dataset composed of 897 sets of ultrasound images and videos. Experimental results show that our model boosts the performance of Benign/Malignant classification, achieving 90.0% in AUC and 81.7% in accuracy.

通过学习静态图像和动态视频的特征，并探索两种模态之间的潜在关系，我们提出了一种模拟放射科医生诊断过程的多模态乳腺肿瘤诊断模型，在在融合多模态特征之前，将动态视频特征在专业放射科医生指导下聚合。在897套乳腺超声图像和视频的数据集上进行实验证实，我们的模型提高了良性/恶性分类的性能，AUC达到90.0%，准确率为81.7%。

利用静态图像和动态视频进行乳腺病变诊断