As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.

本研究解决了表征学习中不同损失函数众多而聚焦的问题，通过提出一个统一的信息论方程，概括了现代损失函数的多样性。研究发现，多个机器学习方法能够最小化两个条件分布之间的KL散度积分，这一新视角为聚类、谱方法和对比学习等提供了深层的信息几何支持，同时通过这一框架开发出新的损失函数，最终实现了在ImageNet-1K上的无监督分类性能提升超过8%。

I-Con：统一的表征学习框架