Multi-modal Learning has attracted widespread attention in medical image
analysis. Using multi-modal data, whole slide images (WSIs) and clinical
information, can improve the performance of deep learning models in the
diagnosis of axillary lymph node metastasis. However, clinical information is
not easy to collect in clinical practice due to privacy concerns, limited
resources, lack of interoperability, etc. Although patient selection can ensure
the training set to have multi-modal data for model development, missing
modality of clinical information can appear during test. This normally leads to
performance degradation, which limits the use of multi-modal models in the
clinic. To alleviate this problem, we propose a bidirectional distillation
framework consisting of a multi-modal branch and a single-modal branch. The
single-modal branch acquires the complete multi-modal knowledge from the
multi-modal branch, while the multi-modal learns the robust features of WSI
from the single-modal. We conduct experiments on a public dataset of Lymph Node
Metastasis in Early Breast Cancer to validate the method. Our approach not only
achieves state-of-the-art performance with an AUC of 0.861 on the test set
without missing data, but also yields an AUC of 0.842 when the rate of missing
modality is 80\%. This shows the effectiveness of the approach in dealing with
multi-modal data and missing modality. Such a model has the potential to
improve treatment decision-making for early breast cancer patients who have
axillary lymph node metastatic status.

基于多模态学习的鉴别性淋巴结转移诊断模型，在临床医学实践中，通过双向蒸馏框架获得全面的多模态知识和稳健的 WSI 特征，有效地处理多模态数据和丢失模态，提高早期乳腺癌患者治疗决策的准确性。

预测腋窝淋巴结转移中的缺失模态下的多模态学习

Multi-modal Learning with Missing Modality in Predicting Axillary Lymph  Node Metastasis

Learning about diagnostic features and related clinical information from
dental radiographs is important for dental research. However, the lack of
expert-annotated data and convenient search tools poses challenges. Our primary
objective is to design a search tool that uses a user's query for oral-related
research. The proposed framework, Contrastive LAnguage Image REtrieval Search
for dental research, Dental CLAIRES, utilizes periapical radiographs and
associated clinical details such as periodontal diagnosis, demographic
information to retrieve the best-matched images based on the text query. We
applied a contrastive representation learning method to find images described
by the user's text by maximizing the similarity score of positive pairs (true
pairs) and minimizing the score of negative pairs (random pairs). Our model
achieved a hit@3 ratio of 96% and a Mean Reciprocal Rank (MRR) of 0.82. We also
designed a graphical user interface that allows researchers to verify the
model's performance with interactions.

提出了一种基于对比语言图像检索搜索的牙科研究搜索工具，称为 Dental CLAIRES，利用牙周尖周照射 X 光片和相关临床细节来根据文本查询检索最匹配的图像。