Out-of-distribution (OOD) detection is critical for safety-sensitive machine
learning applications and has been extensively studied, yielding a plethora of
methods developed in the literature. However, most studies for OOD detection
did not use pre-trained models and trained a backbone from scratch. In recent
years, transferring knowledge from large pre-trained models to downstream tasks
by lightweight tuning has become mainstream for training in-distribution (ID)
classifiers. To bridge the gap between the practice of OOD detection and
current classifiers, the unique and crucial problem is that the samples whose
information networks know often come as OOD input. We consider that such data
may significantly affect the performance of large pre-trained networks because
the discriminability of these OOD data depends on the pre-training algorithm.
Here, we define such OOD data as PT-OOD (Pre-Trained OOD) data. In this paper,
we aim to reveal the effect of PT-OOD on the OOD detection performance of
pre-trained networks from the perspective of pre-training algorithms. To
achieve this, we explore the PT-OOD detection performance of supervised and
self-supervised pre-training algorithms with linear-probing tuning, the most
common efficient tuning method. Through our experiments and analysis, we find
that the low linear separability of PT-OOD in the feature space heavily
degrades the PT-OOD detection performance, and self-supervised models are more
vulnerable to PT-OOD than supervised pre-trained models, even with
state-of-the-art detection methods. To solve this vulnerability, we further
propose a unique solution to large-scale pre-trained models: Leveraging
powerful instance-by-instance discriminative representations of pre-trained
models and detecting OOD in the feature space independent of the ID decision
boundaries. The code will be available via this https URL

揭示预训练模型在预训练算法视角下，外分布数据对外分布检测性能的影响，并提出利用实例间鉴别性特征空间独立于 ID 决策边界的方法解决预训练模型的脆弱性。