We present Integrated Multimodal Perception (IMP), a simple and scalable
multimodal multi-task training and modeling approach. IMP integrates multimodal
inputs including image, video, text, and audio into a single Transformer
encoder with minimal modality-specific components. IMP makes use of a novel
design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
(MoE) for efficient model \& task scaling. We conduct extensive empirical
studies about IMP and reveal the following key insights: 1) performing gradient
descent updates by alternating on diverse heterogeneous modalities, loss
functions, and tasks, while also varying input resolutions, efficiently
improves multimodal understanding. 2) model sparsification with MoE on a single
modality-agnostic encoder substantially improves the performance, outperforming
dense models that use modality-specific encoders or additional fusion layers
and greatly mitigating the conflicts between modalities. IMP achieves
competitive performance on a wide range of downstream tasks including image
classification, video classification, image-text, and video-text retrieval.
Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
new state-of-the-art in zero-shot video classification. Our model achieves
77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
zero-shot classification accuracy, improving the previous state-of-the-art by
+5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
training computational cost.

该研究介绍了一种称为 “Integrated Multimodal Perception（IMP）” 的多模态多任务训练和建模方法，采用转换编码器对图像、视频、文本和音频等多模态输入进行整合处理，在一系列下游任务中实现了竞争性能和提高。在零样本视频分类方面取得了显著的性能提升，达到了此前所未有的最新水平。

交替梯度下降和专家混合模型用于综合多模态感知

Alternating Gradient Descent and Mixture-of-Experts for Integrated  Multimodal Perception

Most methods tackle zero-shot video classification by aligning
visual-semantic representations within seen classes, which limits
generalization to unseen classes. To enhance model generalizability, this paper
presents an end-to-end framework that preserves alignment and uniformity
properties for representations on both seen and unseen classes. Specifically,
we formulate a supervised contrastive loss to simultaneously align
visual-semantic features (i.e., alignment) and encourage the learned features
to distribute uniformly (i.e., uniformity). Unlike existing methods that only
consider the alignment, we propose uniformity to preserve maximal-info of
existing features, which improves the probability that unobserved features fall
around observed data. Further, we synthesize features of unseen classes by
proposing a class generator that interpolates and extrapolates the features of
seen classes. Besides, we introduce two metrics, closeness and dispersion, to
quantify the two properties and serve as new measurements of model
generalizability. Experiments show that our method significantly outperforms
SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51. Code is
available.

该论文提出了一个端到端的框架，用于在被观察和未被观察的类别上均保留对比和一致性特性的视觉 - 语义表示，通过同时进行对齐和鼓励学习特征分布均匀的监督式对比损失来促进模型的泛化能力，实验结果表明该方法在 UCF101 和 HMDB51 上的相对改进分别达到 28.1% 和 27.0%。