We present an approach for multimodal pathology image search, using dynamic
time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed
into a ranked choice voting scheme to retrieve multiplexed immunofluorescent
imaging (mIF) that is most similar to a query H&E slide. Through training the
VAE and applying DTW, we align and compare mIF and H&E slides. Our method
improves differential diagnosis and therapeutic decisions by integrating
morphological H&E data with immunophenotyping from mIF, providing clinicians a
rich perspective of disease states. This facilitates an understanding of the
spatial relationships in tissue samples and could revolutionize the diagnostic
process, enhancing precision and enabling personalized therapy selection. Our
technique demonstrates feasibility using colorectal cancer and healthy tonsil
samples. An exhaustive ablation study was conducted on a search engine designed
to explore the correlation between multiplexed Immunofluorescence (mIF) and
Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map
these distinct modalities into a unified vector space. Despite extreme class
imbalance, the system demonstrated robustness and utility by returning similar
results across various data features, which suggests potential for future use
in multimodal histopathology data analysis.

该研究提出了一种在 VAE 潜在空间中使用动态时间扭曲 (DTW) 进行多模式病理图像搜索的方法，并将其馈送到排名选择投票方案中，以检索与查询 H&E 幻灯片最相似的多重免疫荧光成像 (mIF)。通过训练 VAE 和应用 DTW，我们对齐并比较 mIF 和 H&E 幻灯片，从而融合了形态学 H&E 数据和来自 mIF 的免疫表型，从而提供了丰富的疾病状态视角，可能 彻底改变诊断过程，提高精度并实现个性化治疗选择的能力。