ImageNet was famously created from Flickr image search results. What if we
recreated ImageNet instead by searching the massive LAION dataset based on
image captions alone? In this work, we carry out this counterfactual
investigation. We find that the resulting ImageNet recreation, which we call
LAIONet, looks distinctly unlike the original. Specifically, the intra-class
similarity of images in the original ImageNet is dramatically higher than it is
for LAIONet. Consequently, models trained on ImageNet perform significantly
worse on LAIONet. We propose a rigorous explanation for the discrepancy in
terms of a subtle, yet important, difference in two plausible causal
data-generating processes for the respective datasets, that we support with
systematic experimentation. In a nutshell, searching based on an image caption
alone creates an information bottleneck that mitigates the selection bias
otherwise present in image-based filtering. Our explanation formalizes a
long-held intuition in the community that ImageNet images are stereotypical,
unnatural, and overly simple representations of the class category. At the same
time, it provides a simple and actionable takeaway for future dataset creation
efforts.

通过仅基于图像标题搜索 LAION 数据集对 ImageNet 进行重建，我们发现重建后的数据集 LAIONet 与原始数据集有很大区别，同时提出数据生成过程的微小但重要差异以解释这种不一致性。

ImageNet 与 LAION 的差异

What Makes ImageNet Look Unlike LAION

Deep neural networks trained using a softmax layer at the top and the
cross-entropy loss are ubiquitous tools for image classification. Yet, this
does not naturally enforce intra-class similarity nor inter-class margin of the
learned deep representations. To simultaneously achieve these two goals,
different solutions have been proposed in the literature, such as the pairwise
or triplet losses. However, such solutions carry the extra task of selecting
pairs or triplets, and the extra computational burden of computing and learning
for many combinations of them. In this paper, we propose a plug-and-play loss
term for deep networks that explicitly reduces intra-class variance and
enforces inter-class margin simultaneously, in a simple and elegant geometric
manner. For each class, the deep features are collapsed into a learned linear
subspace, or union of them, and inter-class subspaces are pushed to be as
orthogonal as possible. Our proposed Orthogonal Low-rank Embedding (OL\'E) does
not require carefully crafting pairs or triplets of samples for training, and
works standalone as a classification loss, being the first reported deep metric
learning framework of its kind. Because of the improved margin between features
of different classes, the resulting deep networks generalize better, are more
discriminative, and more robust. We demonstrate improved classification
performance in general object recognition, plugging the proposed loss term into
existing off-the-shelf architectures. In particular, we show the advantage of
the proposed loss in the small data/model scenario, and we significantly
advance the state-of-the-art on the Stanford STL-10 benchmark.

本文提出了一种基于正交低秩嵌入的深度学习损失函数方法，用于同时提高图像分类的内部同类相似度和不同类之间的距离。该方法不需要手动选择训练样本，可以作为一个分类损失函数独立使用，并且可以提高深度网络的泛化能力、辨别能力和鲁棒性，特别是在数据 / 模型较小的情况下，能够显著提高 Stanford STL-10 数据集上的表现。