We present an approach for multimodal pathology image search, using dynamic time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed into a ranked choice voting scheme to retrieve multiplexed immunofluorescent imaging (mIF) that is most similar to a query H&E slide. Through training the VAE and applying DTW, we align and compare mIF and H&E slides. Our method improves differential diagnosis and therapeutic decisions by integrating morphological H&E data with immunophenotyping from mIF, providing clinicians a rich perspective of disease states. This facilitates an understanding of the spatial relationships in tissue samples and could revolutionize the diagnostic process, enhancing precision and enabling personalized therapy selection. Our technique demonstrates feasibility using colorectal cancer and healthy tonsil samples. An exhaustive ablation study was conducted on a search engine designed to explore the correlation between multiplexed Immunofluorescence (mIF) and Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map these distinct modalities into a unified vector space. Despite extreme class imbalance, the system demonstrated robustness and utility by returning similar results across various data features, which suggests potential for future use in multimodal histopathology data analysis.

该研究提出了一种在VAE潜在空间中使用动态时间扭曲(DTW)进行多模式病理图像搜索的方法，并将其馈送到排名选择投票方案中，以检索与查询H&E幻灯片最相似的多重免疫荧光成像(mIF)。通过训练VAE和应用DTW，我们对齐并比较mIF和H&E幻灯片，从而融合了形态学H&E数据和来自mIF的免疫表型，从而提供了丰富的疾病状态视角，可能 彻底改变诊断过程，提高精度并实现个性化治疗选择的能力。

H&E切片和多重免疫荧光图像之间的多模态病理图像搜索