The advent of the Transformer has led to the development of large language
models (LLM), which appear to demonstrate human-like capabilities. To assess
the generality of this class of models and a variety of other base neural
network architectures to multimodal domains, we evaluated and compared their
capacity for multimodal generalization. We introduce a multimodal
question-answer benchmark to evaluate three specific types of
out-of-distribution (OOD) generalization performance: distractor generalization
(generalization in the presence of distractors), systematic compositional
generalization (generalization to new task permutations), and productive
compositional generalization (generalization to more complex tasks structures).
We found that across model architectures (e.g., RNNs, Transformers, Perceivers,
etc.), models with multiple attention layers, or models that leveraged
cross-attention mechanisms between input domains, fared better. Our positive
results demonstrate that for multimodal distractor and systematic
generalization, either cross-modal attention or models with deeper attention
layers are key architectural features required to integrate multimodal inputs.
On the other hand, neither of these architectural features led to productive
generalization, suggesting fundamental limitations of existing architectures
for specific types of multimodal generalization. These results demonstrate the
strengths and limitations of specific architectural components underlying
modern neural models for multimodal reasoning. Finally, we provide Generic COG
(gCOG), a configurable benchmark with several multimodal generalization splits,
for future studies to explore.

评估了多模态泛化的综合性能，发现多层注意力和跨模态注意力是整合多模态输入所需的关键架构特征，但对于有限类型的多模态泛化存在基本限制，强调了现代神经模型在多模态推理方面的优势和局限性。

论多模态推理中神经网络的泛化能力

On the generalization capacity of neural networks during generic  multimodal reasoning

In this paper, we study a novel problem in egocentric action recognition,
which we term as "Multimodal Generalization" (MMG). MMG aims to study how
systems can generalize when data from certain modalities is limited or even
completely missing. We thoroughly investigate MMG in the context of standard
supervised action recognition and the more challenging few-shot setting for
learning new action categories. MMG consists of two novel scenarios, designed
to support security, and efficiency considerations in real-world applications:
(1) missing modality generalization where some modalities that were present
during the train time are missing during the inference time, and (2)
cross-modal zero-shot generalization, where the modalities present during the
inference time and the training time are disjoint. To enable this
investigation, we construct a new dataset MMG-Ego4D containing data points with
video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
derived from Ego4D dataset, but processed and thoroughly re-annotated by human
experts to facilitate research in the MMG problem. We evaluate a diverse array
of models on MMG-Ego4D and propose new methods with improved generalization
ability. In particular, we introduce a new fusion module with modality dropout
training, contrastive-based alignment training, and a novel cross-modal
prototypical loss for better few-shot performance. We hope this study will
serve as a benchmark and guide future research in multimodal generalization
problems. The benchmark and code will be available at
this https URL

该研究论文探讨了在自心视角动作识别中一个新的问题，即 “多模态推广”，研究了系统在某些模态的数据受限或完全缺失时如何进行推广，并构建了一个包含视频、音频和惯性运动传感器模态的数据集。