Research on multi-modal learning dominantly aligns the modalities in a
unified space at training, and only a single one is taken for prediction at
inference. However, for a real machine, e.g., a robot, sensors could be added
or removed at any time. Thus, it is crucial to enable the machine to tackle the
mismatch and unequal-scale problems of modality combinations between training
and inference. In this paper, we tackle these problems from a new perspective:
"Modalities Help Modalities". Intuitively, we present OmniBind, a novel
two-stage learning framework that can achieve any modality combinations and
interaction. It involves teaching data-constrained, a.k.a, student, modalities
to be aligned with the well-trained data-abundant, a.k.a, teacher, modalities.
This subtly enables the adaptive fusion of any modalities to build a unified
representation space for any combinations. Specifically, we propose Cross-modal
Alignment Distillation (CAD) to address the unequal-scale problem between
student and teacher modalities and effectively align student modalities into
the teacher modalities' representation space in stage one. We then propose an
Adaptive Fusion (AF) module to fuse any modality combinations and learn a
unified representation space in stage two. To address the mismatch problem, we
aggregate existing datasets and combine samples from different modalities by
the same semantics. This way, we build the first dataset for training and
evaluation that consists of teacher (image, text) and student (touch, thermal,
event, point cloud, audio) modalities and enables omni-bind for any of them.
Extensive experiments on the recognition task show performance gains over prior
arts by an average of 4.05 % on the arbitrary modality combination setting. It
also achieves state-of-the-art performance for a single modality, e.g., touch,
with a 4.34 % gain.

通过提出 OmniBind 框架，该研究解决了多模态学习中训练和推理阶段模态组合不匹配和尺度不均衡的问题，通过跨模态对齐蒸馏和自适应融合模块，实现了任意模态组合和统一表示空间的学习，具有较高的性能提升。