Addressing missing modalities presents a critical challenge in multimodal
learning. Current approaches focus on developing models that can handle
modality-incomplete inputs during inference, assuming that the full set of
modalities are available for all the data during training. This reliance on
full-modality data for training limits the use of abundant modality-incomplete
samples that are often encountered in practical settings. In this paper, we
propose a robust universal model with modality reconstruction and model
personalization, which can effectively tackle the missing modality at both
training and testing stages. Our method leverages a multimodal masked
autoencoder to reconstruct the missing modality and masked patches
simultaneously, incorporating an innovative distribution approximation
mechanism to fully utilize both modality-complete and modality-incomplete data.
The reconstructed modalities then contributes to our designed data-model
co-distillation scheme to guide the model learning in the presence of missing
modalities. Moreover, we propose a CLIP-driven hyper-network to personalize
partial model parameters, enabling the model to adapt to each distinct missing
modality scenario. Our method has been extensively validated on two brain tumor
segmentation benchmarks. Experimental results demonstrate the promising
performance of our method, which consistently exceeds previous state-of-the-art
approaches under the all-stage missing modality settings with different missing
ratios. Code will be available.

提出了一种具有模态重建和模型个性化的健壮的通用模型，能够在训练和测试阶段有效地处理缺失的模态，通过多模态掩码自编码器重构缺失的模态和掩码补丁，利用创新的分布近似机制充分利用完整和不完整的数据，提出了基于 CLIP 的超网络来个性化模型参数，能够适应不同的缺失模态场景，并在两个脑肿瘤分割基准上得到广泛验证，在不同缺失比例的全阶段缺失模态设置下始终超过先前最先进的方法，代码将可用。

处理所有阶段缺失的形式：朝着具有健壮重建和个性化能力的通用模型

Dealing with All-stage Missing Modality: Towards A Universal Model with  Robust Reconstruction and Personalization

Event cameras sense per-pixel intensity changes and produce asynchronous
event streams with high dynamic range and less motion blur, showing advantages
over conventional cameras. A hurdle of training event-based models is the lack
of large qualitative labeled data. Prior works learning end-tasks mostly rely
on labeled or pseudo-labeled datasets obtained from the active pixel sensor
(APS) frames; however, such datasets' quality is far from rivaling those based
on the canonical images. In this paper, we propose a novel approach, called
\textbf{EvDistill}, to learn a student network on the unlabeled and unpaired
event data (target modality) via knowledge distillation (KD) from a teacher
network trained with large-scale, labeled image data (source modality). To
enable KD across the unpaired modalities, we first propose a bidirectional
modality reconstruction (BMR) module to bridge both modalities and
simultaneously exploit them to distill knowledge via the crafted pairs, causing
no extra computation in the inference. The BMR is improved by the end-tasks and
KD losses in an end-to-end manner. Second, we leverage the structural
similarities of both modalities and adapt the knowledge by matching their
distributions. Moreover, as most prior feature KD methods are uni-modality and
less applicable to our problem, we propose to leverage an affinity graph KD
loss to boost the distillation. Our extensive experiments on semantic
segmentation and object recognition demonstrate that EvDistill achieves
significantly better results than the prior works and KD with only events and
APS frames.

本文提出了一种名为 EvDistill 的新方法，通过知识蒸馏的方式，从大规模的带标签图像数据中训练的教师网络向未标记和未配对的事件数据（目标模态）学习学生网络；我们的方法通过双向模态重建模块来实现教师和学生之间的知识转移，并且采用亲和图知识蒸馏损失来提高知识蒸馏效果。通过实验，我们证明 EvDistill 相比之前的方法和仅使用事件和 APS 帧的知识蒸馏，取得了更好的语义分割和物体识别结果。