Detecting out-of-distribution (OOD) samples is important for deploying
machine learning models in safety-critical applications such as autonomous
driving and robot-assisted surgery. Existing research has mainly focused on
unimodal scenarios on image data. However, real-world applications are
inherently multimodal, which makes it essential to leverage information from
multiple modalities to enhance the efficacy of OOD detection. To establish a
foundation for more realistic Multimodal OOD Detection, we introduce the
first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes
and varying modality combinations. We first evaluate existing unimodal OOD
detection algorithms on MultiOOD, observing that the mere inclusion of
additional modalities yields substantial improvements. This underscores the
importance of utilizing multiple modalities for OOD detection. Based on the
observation of Modality Prediction Discrepancy between in-distribution (ID) and
OOD data, and its strong correlation with OOD performance, we propose the
Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during
training. Moreover, we introduce a novel outlier synthesis method, NP-Mix,
which explores broader feature spaces by leveraging the information from
nearest neighbor classes and complements A2D to strengthen OOD detection
performance. Extensive experiments on MultiOOD demonstrate that training with
A2D and NP-Mix improves existing OOD detection algorithms by a large margin.
Our source code and MultiOOD benchmark are available at
this https URL

通过引入多模态 OOD（Out-of-Distribution）检测基准 MultiOOD，本研究评估了现有的单模态 OOD 检测算法，并提出了 Agree-to-Disagree（A2D）算法和 NP-Mix 异常值合成方法，通过利用多模态信息和探索更广泛的特征空间，显著提升了现有 OOD 检测算法的性能。