The natural world is abundant with concepts expressed via visual, acoustic,
tactile, and linguistic modalities. Much of the existing progress in multimodal
learning, however, focuses primarily on problems where the same set of
modalities are present at train and test time, which makes learning in
low-resource modalities particularly difficult. In this work, we propose
algorithms for cross-modal generalization: a learning paradigm to train a model
that can (1) quickly perform new tasks in a target modality (i.e.
meta-learning) and (2) doing so while being trained on a different source
modality. We study a key research question: how can we ensure generalization
across modalities despite using separate encoders for different source and
target modalities? Our solution is based on meta-alignment, a novel method to
align representation spaces using strongly and weakly paired cross-modal data
while ensuring quick generalization to new tasks across different modalities.
We study this problem on 3 classification tasks: text to image, image to audio,
and text to speech. Our results demonstrate strong performance even when the
new target modality has only a few (1-10) labeled samples and in the presence
of noisy labels, a scenario particularly prevalent in low-resource modalities.

本研究旨在提出一种算法用于跨模态泛化，通过使用强弱配对的跨模态数据对表示空间进行元对齐的新方法，实现在不同模态下快速完成新任务的训练。该算法被运用于三个分类任务，即文本到图像，图像到音频和文本到语音，即使目标模态只有少量（1-10）标记样本和存在噪声标签，该算法的表现也非常强。