A common phenomena confining the representation quality in Self-Supervised
Learning (SSL) is dimensional collapse (also known as rank degeneration), where
the learned representations are mapped to a low dimensional subspace of the
representation space. The State-of-the-Art SSL methods have shown to suffer
from dimensional collapse and fall behind maintaining full rank. Recent
approaches to prevent this problem have proposed using contrastive losses,
regularization techniques, or architectural tricks. We propose WERank, a new
regularizer on the weight parameters of the network to prevent rank
degeneration at different layers of the network. We provide empirical evidence
and mathematical justification to demonstrate the effectiveness of the proposed
regularization method in preventing dimensional collapse. We verify the impact
of WERank on graph SSL where dimensional collapse is more pronounced due to the
lack of proper data augmentation. We empirically demonstrate that WERank is
effective in helping BYOL to achieve higher rank during SSL pre-training and
consequently downstream accuracy during evaluation probing. Ablation studies
and experimental analysis shed lights on the underlying factors behind the
performance gains of the proposed approach.

使用 WERank 作为网络权重参数的正则化方法，在自监督学习中防止降维，提高图表自监督学习的效果。

WERank：用权重规范化实现自监督学习中的排名降级预防

WERank: Towards Rank Degradation Prevention for Self-Supervised Learning  Using Weight Regularization

Sequential recommendation (SR) investigates the dynamic user preferences
modeling and generates the next-item prediction. The next item preference is
typically generated by the affinity between the sequence and item
representations. However, both sequence and item representations suffer from
the rank degeneration issue due to the data sparsity problem. The rank
degeneration issue significantly impairs the representations for SR. This
motivates us to measure how severe is the rank degeneration issue and alleviate
the sequence and item representation rank degeneration issues simultaneously
for SR.
In this work, we theoretically connect the sequence representation
degeneration issue with the item rank degeneration, particularly for short
sequences and cold items. We also identify the connection between the fast
singular value decay phenomenon and the rank collapse issue in transformer
sequence output and item embeddings. We propose the area under the singular
value curve metric to evaluate the severity of the singular value decay
phenomenon and use it as an indicator of rank degeneration. We further
introduce a novel singular spectrum smoothing regularization to alleviate the
rank degeneration on both sequence and item sides, which is the Singular
sPectrum sMoothing for sequential Recommendation (SPMRec). We also establish a
correlation between the ranks of sequence and item embeddings and the rank of
the user-item preference prediction matrix, which can affect recommendation
diversity. We conduct experiments on four benchmark datasets to demonstrate the
superiority of SPMRec over the state-of-the-art recommendation methods,
especially in short sequences. The experiments also demonstrate a strong
connection between our proposed singular spectrum smoothing and recommendation
diversity.

本文研究了顺序推荐中的排名降级问题，并提出了基于奇异值曲线下面积度量和奇异值平滑正则化方法的顺序推荐算法 SPMRec 来解决此问题，该方法在四个基准数据集上的实验结果表明其优于其他推荐算法，尤其在短序列中表现出更好的推荐效果，并可提高推荐的多样性。