In this work, we have worked towards two major goals. Firstly, we have
investigated the importance of Batch Normalisation (BN) layers in a
non-contrastive representation learning framework called Bootstrap Your Own
Latent (BYOL). We conducted several experiments to conclude that BN layers are
not necessary for representation learning in BYOL. Moreover, BYOL only learns
from the positive pairs of images but ignores other semantically similar images
in the same input batch. For the second goal, we have introduced two new loss
functions to determine the semantically similar pairs in the same input batch
of images and reduce the distance between their representations. These loss
functions are Cross-Cosine Similarity Loss (CCSL) and Cross-Sigmoid Similarity
Loss (CSSL). Using the proposed loss functions, we are able to surpass the
performance of Vanilla BYOL (71.04%) by training the BYOL framework using CCSL
loss (76.87%) on the STL10 dataset. BYOL trained using CSSL loss performs
comparably with Vanilla BYOL.

研究 BYOL 中批标准化层的重要性，提出使用 CCSL 和 CSSL 进行相似图片对比的方法来改进 BYOL 的表征学习效果。在 STL10 数据集上使用 CSSL loss 训练 BYOL 框架可以取得较好的效果。

基于自举潜变量的强鲁棒性表示学习的自标注优化

Self-Labeling Refinement for Robust Representation Learning with Bootstrap Your Own Latent

Inspired by the recent progress in self-supervised learning for computer
vision that generates supervision using data augmentations, we explore a new
general-purpose audio representation learning approach. We propose learning
general-purpose audio representation from a single audio segment without
expecting relationships between different time segments of audio samples. To
implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for
Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method
based on BYOL for learning general-purpose audio representation. Unlike most
previous audio self-supervised learning methods that rely on agreement of
vicinity audio segments or disagreement of remote ones, BYOL-A creates
contrasts in an augmented audio segment pair derived from a single audio
segment. With a combination of normalization and augmentation techniques,
BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive
ablation studies also clarified the contribution of each component and their
combinations.

本研究提出了一种基于自我监督学习的通用音频表示学习方法，采用 Bootstrap Your Own Latent (BYOL) for Audio 方法，通过标准化和数据增强技术，从单个音频段中学习音频表示，在各种下游任务中实现了最先进的结果。