Deep Learning models have been successfully utilized to extract clinically
actionable insights from routinely available histology data. Generally, these
models require annotations performed by clinicians, which are scarce and costly
to generate. The emergence of self-supervised learning (SSL) methods remove
this barrier, allowing for large-scale analyses on non-annotated data. However,
recent SSL approaches apply increasingly expansive model architectures and
larger datasets, causing the rapid escalation of data volumes, hardware
prerequisites, and overall expenses, limiting access to these resources to few
institutions. Therefore, we investigated the complexity of contrastive SSL in
computational pathology in relation to classification performance with the
utilization of consumer-grade hardware. Specifically, we analyzed the effects
of adaptations in data volume, architecture, and algorithms on downstream clas-
sification tasks, emphasizing their impact on computational resources. We
trained breast cancer foundation models on a large public patient cohort and
validated them on various downstream classification tasks in a weakly
supervised manner on two external public patient cohorts. Our experiments
demonstrate that we can improve downstream classification performance whilst
reducing SSL training duration by 90%. In summary, we propose a set of
adaptations which enable the utilization of SSL in computational pathology in
non-resource abundant environments.

我们提出了一套适应性方法，可以在资源有限的环境下利用半监督自学习来进行计算病理学研究，从而提高下游分类性能并缩短自学习训练时间。

简化自监督学习复杂度提高计算病理学弱监督分类性能

Reducing self-supervised learning complexity improves weakly-supervised  classification performance in computational pathology

Sentence Representation Learning (SRL) is a crucial task in Natural Language
Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently
a mainstream approach. However, the reasons behind its remarkable effectiveness
remain unclear. Specifically, in other research fields, contrastive SSL shares
similarities in both theory and practical performance with non-contrastive SSL
(e.g., alignment & uniformity, Barlow Twins, and VICReg). However, in SRL,
contrastive SSL outperforms non-contrastive SSL significantly. Therefore, two
questions arise: First, what commonalities enable various contrastive losses to
achieve superior performance in SRL? Second, how can we make non-contrastive
SSL, which is similar to contrastive SSL but ineffective in SRL, effective? To
address these questions, we start from the perspective of gradients and
discover that four effective contrastive losses can be integrated into a
unified paradigm, which depends on three components: the Gradient Dissipation,
the Weight, and the Ratio. Then, we conduct an in-depth analysis of the roles
these components play in optimization and experimentally demonstrate their
significance for model performance. Finally, by adjusting these components, we
enable non-contrastive SSL to achieve outstanding performance in SRL.

自监督学习在自然语言处理中的关键任务是句子表示学习，其中对比自监督学习是目前的主流方法。然而，其显著有效性背后的原因仍不清楚。本研究从梯度的角度出发，发现四种有效的对比损失可以集成到一个统一的范式中，这依赖于梯度消散、权重和比率这三个组成部分，并通过深入分析这些组件在优化中的作用，实验证明它们对模型性能的重要性，并通过调整这些组件使得非对比自监督学习在句子表示学习中取得出色的性能。

构建更好的对比句子表示学习方法：梯度的统一范式

Towards Better Understanding of Contrastive Sentence Representation  Learning: A Unified Paradigm for Gradient

We propose a novel theoretical framework to understand contrastive
self-supervised learning (SSL) methods that employ dual pairs of deep ReLU
networks (e.g., SimCLR). First, we prove that in each SGD update of SimCLR with
various loss functions, including simple contrastive loss, soft Triplet loss
and InfoNCE loss, the weights at each layer are updated by a \emph{covariance
operator} that specifically amplifies initial random selectivities that vary
across data samples but survive averages over data augmentations. To further
study what role the covariance operator plays and which features are learned in
such a process, we model data generation and augmentation processes through a
\emph{hierarchical latent tree model} (HLTM) and prove that the hidden neurons
of deep ReLU networks can learn the latent variables in HLTM, despite the fact
that the network receives \emph{no direct supervision} from these unobserved
latent variables. This leads to a provable emergence of hierarchical features
through the amplification of initially random selectivities through contrastive
SSL. Extensive numerical studies justify our theoretical findings. Code is
released in this https URL

提出了一种新的理论框架来理解对比自监督学习方法，该方法采用双重 ReLU 网络（例如 SimCLR）。我们证明了每个 SGD 更新中的权重是由协方差算子更新的，并进一步说明了协方差算子的作用和应用。通过建立层次潜变量树模型并证明深度 ReLU 网络的隐藏层学习了该模型的潜变量，我们总结了通过对比自监督学习的初始随机选择法放大而形成的层次化特征。