Learning multimodal representations involves integrating information from
multiple heterogeneous sources of data. In order to accelerate progress towards
understudied modalities and tasks while ensuring real-world robustness, we
release MultiZoo, a public toolkit consisting of standardized implementations
of > 20 core multimodal algorithms and MultiBench, a large-scale benchmark
spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Together, these provide an automated end-to-end machine learning pipeline that
simplifies and standardizes data loading, experimental setup, and model
evaluation. To enable holistic evaluation, we offer a comprehensive methodology
to assess (1) generalization, (2) time and space complexity, and (3) modality
robustness. MultiBench paves the way towards a better understanding of the
capabilities and limitations of multimodal models, while ensuring ease of use,
accessibility, and reproducibility. Our toolkits are publicly available, will
be regularly updated, and welcome inputs from the community.

MultiZoo 和 MultiBench 提供了一个自动化的机器学习流水线，能够简化和标准化数据加载、实验设置和模型评估，这些对于研究者们了解多模态模型的能力和局限性是很有帮助的。

MultiZoo 和 MultiBench：多模态深度学习的标准化工具包

MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep  Learning

Learning multimodal representations involves integrating information from
multiple heterogeneous sources of data. It is a challenging yet crucial area
with numerous real-world applications in multimedia, affective computing,
robotics, finance, human-computer interaction, and healthcare. Unfortunately,
multimodal research has seen limited resources to study (1) generalization
across domains and modalities, (2) complexity during training and inference,
and (3) robustness to noisy and missing modalities. In order to accelerate
progress towards understudied modalities and tasks while ensuring real-world
robustness, we release MultiBench, a systematic and unified large-scale
benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6
research areas. MultiBench provides an automated end-to-end machine learning
pipeline that simplifies and standardizes data loading, experimental setup, and
model evaluation. To enable holistic evaluation, MultiBench offers a
comprehensive methodology to assess (1) generalization, (2) time and space
complexity, and (3) modality robustness. MultiBench introduces impactful
challenges for future research, including scalability to large-scale multimodal
datasets and robustness to realistic imperfections. To accompany this
benchmark, we also provide a standardized implementation of 20 core approaches
in multimodal learning. Simply applying methods proposed in different research
areas can improve the state-of-the-art performance on 9/15 datasets. Therefore,
MultiBench presents a milestone in unifying disjoint efforts in multimodal
research and paves the way towards a better understanding of the capabilities
and limitations of multimodal models, all the while ensuring ease of use,
accessibility, and reproducibility. MultiBench, our standardized code, and
leaderboards are publicly available, will be regularly updated, and welcomes
inputs from the community.

MultiBench 是一个系统性和统一化的基准测试，跨越 15 个数据集，10 种模态，20 种预测任务和 6 个研究领域。MultiBench 提供自动化的端到端机器学习流程，简化和标准化数据加载，实验设置和模型评估。随着大量的实践证明，不同研究领域提出的方法可以改善 9/15 数据集的最先进性能。MultiBench 处理了跨模态一般化，复杂性和健壮性的问题，对于未来研究具有重要的挑战性，包括可扩展性到大规模多模态数据集和对现实缺陷的健壮性。