Imaging modalities such as Computed Tomography (CT) and Positron Emission
Tomography (PET) are key in cancer detection, inspiring Deep Neural Networks
(DNN) models that merge these scans for tumor segmentation. When both CT and
PET scans are available, it is common to combine them as two channels of the
input to the segmentation model. However, this method requires both scan types
during training and inference, posing a challenge due to the limited
availability of PET scans, thereby sometimes limiting the process to CT scans
only. Hence, there is a need to develop a flexible DNN architecture that can be
trained/updated using only CT scans but can effectively utilize PET scans when
they become available. In this work, we propose a parameter-efficient
multi-modal adaptation (PEMMA) framework for lightweight upgrading of a
transformer-based segmentation model trained only on CT scans to also
incorporate PET scans. The benefits of the proposed approach are two-fold.
Firstly, we leverage the inherent modularity of the transformer architecture
and perform low-rank adaptation (LoRA) of the attention weights to achieve
parameter-efficient adaptation. Secondly, since the PEMMA framework attempts to
minimize cross modal entanglement, it is possible to subsequently update the
combined model using only one modality, without causing catastrophic forgetting
of the other modality. Our proposed method achieves comparable results with the
performance of early fusion techniques with just 8% of the trainable
parameters, especially with a remarkable +28% improvement on the average dice
score on PET scans when trained on a single modality.

提出了一种适用于基于 CT 扫描模型的轻量级升级的参数高效多模态适应（PEMMA）框架，以将 PET 扫描纳入其中，通过将注意力权重进行低秩适应（LoRA）实现参数高效调整，同时最小化交叉模态耦合，使单一模态下训练和更新联合模型成为可能，并且在仅训练单一模态时，在可训练参数仅为 8% 的情况下，实现了与早期融合技术相当的结果，并在 PET 扫描的平均 Dice 分数上取得了显著的 + 28% 改善。

PEMMA: 用于医学图像分割的参数高效多模态适应性

PEMMA: Parameter-Efficient Multi-Modal Adaptation for Medical Image  Segmentation

As the size and complexity of data continue to increase, the need for
efficient and effective analysis methods becomes ever more crucial.
Tensorization, the process of converting 2-dimensional datasets into
multidimensional structures, has emerged as a promising approach for multiway
analysis methods. This paper explores the steps involved in tensorization,
multidimensional data sources, various multiway analysis methods employed, and
the benefits of these approaches. A small example of Blind Source Separation
(BSS) is presented comparing 2-dimensional algorithms and a multiway algorithm
in Python. Results indicate that multiway analysis is more expressive.
Additionally, tensorization techniques aid in compressing deep learning models
by reducing the number of required parameters while enhancing the expression of
relationships across dimensions. A survey of the multi-away analysis methods
and integration with various Deep Neural Networks models is presented using
case studies in different domains.

本文研究了通过将二维数据集转化为多维结构的张量化过程，以及相关的多维分析方法，分析了这些方法的优势。通过比较二维算法和多维算法在 Python 中的一个盲源分离示例，结果表明多维分析更具表达力。此外，张量化技术有助于在减少所需参数数量的同时增强深度学习模型之间的关系表达，并对不同领域的案例研究进行了多维分析方法的调查和与各种深度神经网络模型的集成。

张量化：为多方分析和张量深度神经网络创建和利用多维数据集 -- Python 教程和调查

Tensorization: Creating and Utilising Multidimensional Datasets for  Multiway Analysis and Tensorised Deep Neural Networks -- Python Tutorial and  Survey