This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.

本文介绍了相对预测编码（RPC），一种新的对比表示学习目标，保持了训练稳定性，小批量大小敏感性和下游任务性能之间的良好平衡。RPC的成功关键在于两个方面。首先，RPC引入了相对参数来调整目标，使其具有界限性和低方差。第二，RPC不包含对数和指数得分函数，这是以前对比目标中训练不稳定性的主要原因。我们在基准视觉和语音自监督学习任务上经验证实了RPC的有效性。最后，我们将RPC与相互信息（MI）估计联系起来，显示RPC可以用于低方差的MI估计。

使用相对预测编码的自监督表示学习