Self-distillation methods using Siamese networks are popular for
self-supervised pre-training. DINO is one such method based on a cross-entropy
loss between $K$-dimensional probability vectors, obtained by applying a
softmax function to the dot product between representations and learnt
prototypes. Given the fact that the learned representations are
$L^2$-normalized, we show that DINO and its derivatives, such as iBOT, can be
interpreted as a mixture model of von Mises-Fisher components. With this
interpretation, DINO assumes equal precision for all components when the
prototypes are also $L^2$-normalized. Using this insight we propose DINO-vMF,
that adds appropriate normalization constants when computing the cluster
assignment probabilities. Unlike DINO, DINO-vMF is stable also for the larger
ViT-Base model with unnormalized prototypes. We show that the added flexibility
of the mixture model is beneficial in terms of better image representations.
The DINO-vMF pre-trained model consistently performs better than DINO on a
range of downstream tasks. We obtain similar improvements for iBOT-vMF vs iBOT
and thereby show the relevance of our proposed modification also for other
methods derived from DINO.

DINO-vMF 使用混合模型的方法改进了 DINO 的集群分配概率计算，对于较大的 ViT-Base 模型也更加稳定且在各种下游任务中表现更好，提供了更好的图像表示。

DINO 作为 von Mises-Fisher 混合模型

DINO as a von Mises-Fisher mixture model

Sequential recommendation methods play a pivotal role in modern
recommendation systems. A key challenge lies in accurately modeling user
preferences in the face of data sparsity. To tackle this challenge, recent
methods leverage contrastive learning (CL) to derive self-supervision signals
by maximizing the mutual information of two augmented views of the original
user behavior sequence. Despite their effectiveness, CL-based methods encounter
a limitation in fully exploiting self-supervision signals for users with
limited behavior data, as users with extensive behaviors naturally offer more
information. To address this problem, we introduce a novel learning paradigm,
named Online Self-Supervised Self-distillation for Sequential Recommendation
($S^4$Rec), effectively bridging the gap between self-supervised learning and
self-distillation methods. Specifically, we employ online clustering to
proficiently group users by their distinct latent intents. Additionally, an
adversarial learning strategy is utilized to ensure that the clustering
procedure is not affected by the behavior length factor. Subsequently, we
employ self-distillation to facilitate the transfer of knowledge from users
with extensive behaviors (teachers) to users with limited behaviors (students).
Experiments conducted on four real-world datasets validate the effectiveness of
the proposed method\footnote{Code is available at this https URL

在线自监督自蒸馏的顺序推荐方法（S^4Rec）应用于现代推荐系统中，通过在线聚类和对抗学习策略来有效地处理数据稀疏性和用户行为数据有限性的挑战，并利用自监督学习和自蒸馏方法实现知识传递。