Multimodal Deep Learning enhances decision-making by integrating diverse
information sources, such as texts, images, audio, and videos. To develop
trustworthy multimodal approaches, it is essential to understand how
uncertainty impacts these models. We introduce LUMA, a unique benchmark
dataset, featuring audio, image, and textual data from 50 classes, for learning
from uncertain and multimodal data. It extends the well-known CIFAR 10/100
dataset with audio samples extracted from three audio corpora, and text data
generated using the Gemma-7B Large Language Model (LLM). The LUMA dataset
enables the controlled injection of varying types and degrees of uncertainty to
achieve and tailor specific experiments and benchmarking initiatives. LUMA is
also available as a Python package including the functions for generating
multiple variants of the dataset with controlling the diversity of the data,
the amount of noise for each modality, and adding out-of-distribution samples.
A baseline pre-trained model is also provided alongside three uncertainty
quantification methods: Monte-Carlo Dropout, Deep Ensemble, and Reliable
Conflictive Multi-View Learning. This comprehensive dataset and its tools are
intended to promote and support the development and benchmarking of trustworthy
and robust multimodal deep learning approaches.

多模态深度学习通过整合文本、图像、音频和视频等多元信息源增强决策能力。为了开发可靠的多模态方法，了解不确定性如何影响这些模型至关重要。我们介绍了 LUMA 数据集，它是一个独特的基准数据集，其中包含来自 50 个类别的音频、图像和文本数据，用于从不确定和多模态数据中学习。该数据集扩展了著名的 CIFAR 10/100 数据集，其中包含从三个音频语料库中提取的音频样本，以及使用 Gemma-7B 大型语言模型 (LLM) 生成的文本数据。LUMA 数据集可以在实验和基准测试活动中有计划地注入不同类型和程度的不确定性，以实现个性化的实验和评估。LUMA 还作为一个 Python 软件包提供，其中包含用于生成数据集的多个变种的函数，可以控制数据的多样性、每种模态的噪声量以及添加超出分布范围的样本。同时还提供了一个基线预训练模型以及三种不确定性量化方法：Monte-Carlo Dropout、Deep Ensemble 和 Reliable Conflictive Multi-View Learning。这个全面的数据集及其工具旨在推动和支持可靠和鲁棒的多模态深度学习方法的发展和基准测试。