This study proposes a multi-modal fusion framework Multitrans based on the
Transformer architecture and self-attention mechanism. This architecture
combines the study of non-contrast computed tomography (NCCT) images and
discharge diagnosis reports of patients undergoing stroke treatment, using a
variety of methods based on Transformer architecture approach to predicting
functional outcomes of stroke treatment. The results show that the performance
of single-modal text classification is significantly better than single-modal
image classification, but the effect of multi-modal combination is better than
any single modality. Although the Transformer model only performs worse on
imaging data, when combined with clinical meta-diagnostic information, both can
learn better complementary information and make good contributions to
accurately predicting stroke treatment effects..

本研究提出了一种基于 Transformer 架构和自注意机制的多模态融合框架 Multitrans，通过将非对比度计算机断层成像（NCCT）图像和中风治疗患者的出院诊断报告结合起来，使用一种基于 Transformer 架构的多种方法来预测中风治疗的功能结果。结果表明，单模态文本分类的性能明显优于单模态图像分类，但多模态组合的效果优于任何单模态。尽管 Transformer 模型在成像数据上的性能不佳，但当与临床元诊断信息结合时，两者都能学习到更好的互补信息，并为准确预测中风治疗效果做出良好贡献。

基于 Transformer 的多模态脑卒中治疗分类结果预测

Transformer-Based Classification Outcome Prediction for Multimodal  Stroke Treatment

Accurate automated extraction of brain vessel centerlines from CTA images
plays an important role in diagnosis and therapy of cerebrovascular diseases,
such as stroke. However, this task remains challenging due to the complex
cerebrovascular structure, the varying imaging quality, and vessel pathology
effects. In this paper, we consider automatic lumen segmentation generation
without additional annotation effort by physicians and more effective use of
the generated lumen segmentation for improved centerline extraction
performance. We propose an automated framework for brain vessel centerline
extraction from CTA images. The framework consists of four major components:
(1) pre-processing approaches that register CTA images with a CT atlas and
divide these images into input patches, (2) lumen segmentation generation from
annotated vessel centerlines using graph cuts and robust kernel regression, (3)
a dual-branch topology-aware UNet (DTUNet) that can effectively utilize the
annotated vessel centerlines and the generated lumen segmentation through a
topology-aware loss (TAL) and its dual-branch design, and (4) post-processing
approaches that skeletonize the predicted lumen segmentation. Extensive
experiments on a multi-center dataset demonstrate that the proposed framework
outperforms state-of-the-art methods in terms of average symmetric centerline
distance (ASCD) and overlap (OV). Subgroup analyses further suggest that the
proposed framework holds promise in clinical applications for stroke treatment.
Code is publicly available at this https URL

提出了一个自动化的框架来从 CTA 图像中提取脑血管中心线，该框架包括预处理方法、基于图割和稳健核回归的管腔分割、采用拓扑感知损失和双分支设计的双分支拓扑感知 UNet (DTUNet)，以及骨架化预测的管腔分割的后处理方法，实验证明该框架在平均对称中心线距离 (ASCD) 和重叠度 (OV) 方面优于现有方法，并且子组分析进一步表明该框架在中风治疗的临床应用中有潜力。