Recent research at CHU Sainte Justine's Pediatric Critical Care Unit (PICU)
has revealed that traditional machine learning methods, such as semi-supervised
label propagation and K-nearest neighbors, outperform Transformer-based models
in artifact detection from PPG signals, mainly when data is limited. This study
addresses the underutilization of abundant unlabeled data by employing
self-supervised learning (SSL) to extract latent features from these data,
followed by fine-tuning on labeled data. Our experiments demonstrate that SSL
significantly enhances the Transformer model's ability to learn
representations, improving its robustness in artifact classification tasks.
Among various SSL techniques, including masking, contrastive learning, and DINO
(self-distillation with no labels)-contrastive learning exhibited the most
stable and superior performance in small PPG datasets. Further, we delve into
optimizing contrastive loss functions, which are crucial for contrastive SSL.
Inspired by InfoNCE, we introduce a novel contrastive loss function that
facilitates smoother training and better convergence, thereby enhancing
performance in artifact classification. In summary, this study establishes the
efficacy of SSL in leveraging unlabeled data, particularly in enhancing the
capabilities of the Transformer model. This approach holds promise for broader
applications in PICU environments, where annotated data is often limited.

该研究在儿童重症监护单元 (CHU Sainte Justine's PICU) 的最新研究中发现，在 PPG 信号的伪迹检测方面，传统的机器学习方法（如半监督标签传播和 K 最近邻算法）在数据有限时效果优于基于 Transformer 模型的方法。本研究通过自我监督学习 (Self-Supervised Learning, SSL) 从未标记数据中提取潜在特征，并结合有标签数据进行微调，证明了 SSL 能够显著增强 Transformer 模型的表示学习能力，在伪迹分类任务中提高了其稳健性。其中，对比学习 (contrastive learning) 是众多 SSL 技术中表现最稳定且性能最优的技术，特别适用于小规模的 PPG 数据集。此外，研究还详细探讨了对比损失函数优化的问题，该函数对于对比自我监督学习至关重要。受 InfoNCE 的启发，我们引入了一种新的对比损失函数，有助于训练更平稳、收敛更好，从而提高伪迹分类的性能。总之，该研究证实了 SSL 在利用未标记数据方面的效果，特别是在增强 Transformer 模型能力方面，并有望在注释数据有限的 PICU 环境中广泛应用。

自监督学习增强变压器在 PPG 信号伪迹检测中的鲁棒性和效能

Boosting Transformer's Robustness and Efficacy in PPG Signal Artifact  Detection with Self-Supervised Learning

In standard supervised machine learning, it is necessary to provide a label
for every input in the data. While raw data in many application domains is
easily obtainable on the Internet, manual labelling of this data is
prohibitively expensive. To circumvent this issue, contrastive learning methods
produce low-dimensional vector representations (also called features) of
high-dimensional inputs on large unlabelled datasets. This is done by training
with a contrastive loss function, which enforces that similar inputs have high
inner product and dissimilar inputs have low inner product in the feature
space. Rather than annotating each input individually, it suffices to define a
means of sampling pairs of similar and dissimilar inputs. Contrastive features
can then be fed as inputs to supervised learning systems on much smaller
labelled datasets to obtain high accuracy on end tasks of interest.
The goal of this thesis is to provide an overview of the current theoretical
understanding of contrastive learning, specifically as it pertains to the
minimizers of contrastive loss functions and their relationship to prior
methods for learning features from unlabelled data. We highlight popular
contrastive loss functions whose minimizers implicitly approximate a positive
semidefinite (PSD) kernel. The latter is a well-studied object in functional
analysis and learning theory that formalizes a notion of similarity between
elements of a space. PSD kernels provide an implicit definition of features
through the theory of reproducing kernel Hilbert spaces.

通过对无标签数据进行对比学习，可以生成低维的特征向量表示，这些特征向量可以作为输入来提高有标签数据上的监督学习系统的准确性，并探讨了对比损失函数的最小化器及其与以往学习无标签数据方法的关系。

对比学习作为核逼近

Contrastive Learning as Kernel Approximation

Fitness functions map large combinatorial spaces of biological sequences to
properties of interest. Inferring these multimodal functions from experimental
data is a central task in modern protein engineering. Global epistasis models
are an effective and physically-grounded class of models for estimating fitness
functions from observed data. These models assume that a sparse latent function
is transformed by a monotonic nonlinearity to emit measurable fitness. Here we
demonstrate that minimizing contrastive loss functions, such as the
Bradley-Terry loss, is a simple and flexible technique for extracting the
sparse latent function implied by global epistasis. We argue by way of a
fitness-epistasis uncertainty principle that the nonlinearities in global
epistasis models can produce observed fitness functions that do not admit
sparse representations, and thus may be inefficient to learn from observations
when using a Mean Squared Error (MSE) loss (a common practice). We show that
contrastive losses are able to accurately estimate a ranking function from
limited data even in regimes where MSE is ineffective. We validate the
practical utility of this insight by showing contrastive loss functions result
in consistently improved performance on benchmark tasks.

通过对全局阴性相互作用模型使用对比损失函数技术，文章提出并验证该技术可以在有限的数据量下，精确地估计排名函数，并具有在常规均方误差损失函数无效的情况下提高性能的实际效用。

对比损失作为全局上表观互作的广义模型

Contrastive losses as generalized models of global epistasis

To train image-caption retrieval (ICR) methods, contrastive loss functions
are a common choice for optimization functions. Unfortunately, contrastive ICR
methods are vulnerable to learning shortcuts: decision rules that perform well
on the training data but fail to transfer to other testing conditions. We
introduce an approach to reduce shortcut feature representations for the ICR
task: latent target decoding (LTD). We add an additional decoder to the
learning framework to reconstruct the input caption, which prevents the image
and caption encoder from learning shortcut features. Instead of reconstructing
input captions in the input space, we decode the semantics of the caption in a
latent space. We implement the LTD objective as an optimization constraint, to
ensure that the reconstruction loss is below a threshold value while primarily
optimizing for the contrastive loss. Importantly, LTD does not depend on
additional training data or expensive (hard) negative mining strategies. Our
experiments show that, unlike reconstructing the input caption, LTD reduces
shortcut learning and improves generalizability by obtaining higher recall@k
and r-precision scores. Additionally, we show that the evaluation scores
benefit from implementing LTD as an optimization constraint instead of a dual
loss.

采用潜在目标解码 (LTD) 方法作为优化约束来降低图像 - 标题检索 (ICR) 方法的学习捷径，以提高其一般化能力和召回率。