We introduce a novel and general loss function, called Symmetric Contrastive
(Sy-CON) loss, for effective continual self-supervised learning (CSSL). We
first argue that the conventional loss form of continual learning which
consists of single task-specific loss (for plasticity) and a regularizer (for
stability) may not be ideal for contrastive loss based CSSL that focus on
representation learning. Our reasoning is that, in contrastive learning based
methods, the task-specific loss would suffer from decreasing diversity of
negative samples and the regularizer may hinder learning new distinctive
representations. To that end, we propose Sy-CON that consists of two losses
(one for plasticity and the other for stability) with symmetric dependence on
current and past models' negative sample embeddings. We argue our model can
naturally find good trade-off between the plasticity and stability without any
explicit hyperparameter tuning. We validate the effectiveness of our approach
through extensive experiments, demonstrating that MoCo-based implementation of
Sy-CON loss achieves superior performance compared to other state-of-the-art
CSSL methods.

引入了一种新型通用损失函数 Symmetric Contrastive (Sy-CON) loss 用于有效地连续自监督学习 (CSSL).

Sy-CON: 对称对比损失用于连续自监督表示学习

Sy-CON: Symmetric Contrastive Loss for Continual Self-Supervised  Representation Learning

Most state-of-the-art self-supervised speaker verification systems rely on a
contrastive-based objective function to learn speaker representations from
unlabeled speech data. We explore different ways to improve the performance of
these methods by: (1) revisiting how positive and negative pairs are sampled
through a "symmetric" formulation of the contrastive loss; (2) introducing
margins similar to AM-Softmax and AAM-Softmax that have been widely adopted in
the supervised setting. We demonstrate the effectiveness of the symmetric
contrastive loss which provides more supervision for the self-supervised task.
Moreover, we show that Additive Margin and Additive Angular Margin allow
reducing the overall number of false negatives and false positives by improving
speaker separability. Finally, by combining both techniques and training a
larger model we achieve 7.50% EER and 0.5804 minDCF on the VoxCeleb1 test set,
which outperforms other contrastive self supervised methods on speaker
verification.

我们通过 “对称” 形式的对比损失重新思考正负样本对的采样方式，并引入了在监督设置中被广泛采用的 AM-Softmax 和 AAM-Softmax 等边缘，展示了对称对比损失的有效性，以及 Additive Margin 和 Additive Angular Margin 能够通过改善讲话人可分性来减少错误识别。最终，通过结合这两种技术并训练一个更大的模型，我们在 VoxCeleb1 测试集上取得了 7.50% 的 EER 和 0.5804 的 minDCF，优于其他对比自监督方法在讲话人验证上的表现。