Including information from additional spectral bands (e.g., near-infrared)
can improve deep learning model performance for many vision-oriented tasks.
There are many possible ways to incorporate this additional information into a
deep learning model, but the optimal fusion strategy has not yet been
determined and can vary between applications. At one extreme, known as "early
fusion," additional bands are stacked as extra channels to obtain an input
image with more than three channels. At the other extreme, known as "late
fusion," RGB and non-RGB bands are passed through separate branches of a deep
learning model and merged immediately before a final classification or
segmentation layer. In this work, we characterize the performance of a suite of
multispectral deep learning models with different fusion approaches, quantify
their relative reliance on different input bands and evaluate their robustness
to naturalistic image corruptions affecting one or more input channels.

通过研究多光谱深度学习模型的各种融合方法，我们评估了这些模型对不同输入波段的相对依赖性以及对影响一个或多个输入通道的自然图像损坏的稳健性。

多光谱深度神经网络的鲁棒性和可解释性对架构的影响

Impact of architecture on robustness and interpretability of  multispectral deep neural networks

Labeling of multivariate biomedical time series data is a laborious and
expensive process. Self-supervised contrastive learning alleviates the need for
large, labeled datasets through pretraining on unlabeled data. However, for
multivariate time series data the set of input channels often varies between
applications, and most existing work does not allow for transfer between
datasets with different sets of input channels. We propose learning one encoder
to operate on all input channels individually. We then use a message passing
neural network to extract a single representation across channels. We
demonstrate the potential of this method by pretraining our network on a
dataset with six EEG channels and finetuning on a dataset with two different
EEG channels. We compare networks with and without the message passing neural
network across different contrastive loss functions. We show that our method
combined with the TS2Vec loss outperforms all other methods in most settings.

通过在多通道生物医学时间序列数据上进行自我监督对比学习，使用消息传递神经网络提取跨通道的单一表示，并结合 TS2Vec 损失函数，我们的方法在大多数设置中优于其他方法。