In this paper, we study contrastive learning from an optimization
perspective, aiming to analyze and address a fundamental issue of existing
contrastive learning methods that either rely on a large batch size or a large
dictionary of feature vectors. We consider a global objective for contrastive
learning, which contrasts each positive pair with all negative pairs for an
anchor point. From the optimization perspective, we explain why existing
methods such as SimCLR require a large batch size in order to achieve a
satisfactory result. In order to remove such requirement, we propose a
memory-efficient Stochastic Optimization algorithm for solving the Global
objective of Contrastive Learning of Representations, named SogCLR. We show
that its optimization error is negligible under a reasonable condition after a
sufficient number of iterations or is diminishing for a slightly different
global contrastive objective. Empirically, we demonstrate that SogCLR with
small batch size (e.g., 256) can achieve similar performance as SimCLR with
large batch size (e.g., 8192) on self-supervised learning task on ImageNet-1K.
We also attempt to show that the proposed optimization technique is generic and
can be applied to solving other contrastive losses, e.g., two-way contrastive
losses for bimodal contrastive learning. The proposed method is implemented in
our open-sourced library LibAUC (www.libauc.org).

本文从优化的角度研究对比学习，提出了一种内存效率高的随机优化算法 ——SogCLR，可解决现有对比学习方法依赖大批量数据或特征向量词典的问题，并在自监督学习任务中取得了与 SimCLR 相似的性能。

全局对比学习中可证明的随机优化：小批量不会影响性能

Provable Stochastic Optimization for Global Contrastive Learning: Small  Batch Does Not Harm Performance

We present a new model for prediction markets, in which we use risk measures
to model agents and introduce a market maker to describe the trading process.
This specific choice on modelling tools brings us mathematical convenience. The
analysis shows that the whole market effectively approaches a global objective,
despite that the market is designed such that each agent only cares about its
own goal. Additionally, the market dynamics provides a sensible algorithm for
optimising the global objective. An intimate connection between machine
learning and our markets is thus established, such that we could 1) analyse a
market by applying machine learning methods to the global objective, and 2)
solve machine learning problems by setting up and running certain markets.

本文提出一种新的预测市场模型，使用风险度量来建模市场中的代理人，并引入市场制造商来描述交易过程，分析表明，整个市场有效地达到了全局目标，同时建立了机器学习与市场之间的密切联系。