Support vector clustering is an important clustering method. However, it
suffers from a scalability issue due to its computational expensive cluster
assignment step. In this paper we accelertate the support vector clustering via
spectrum-preserving data compression. Specifically, we first compress the
original data set into a small amount of spectrally representative aggregated
data points. Then, we perform standard support vector clustering on the
compressed data set. Finally, we map the clustering results of the compressed
data set back to discover the clusters in the original data set. Our extensive
experimental results on real-world data set demonstrate dramatically speedups
over standard support vector clustering without sacrificing clustering quality.

该研究通过光谱保留数据压缩的方式来加速支持向量聚类，并取得了显著的速度提升和聚类质量保持。

谱保持数据压缩加速支持向量聚类

Accelerate Support Vector Clustering via Spectrum-Preserving Data  Compression?

Vision Transformer (ViT) has recently demonstrated promise in computer vision
problems. However, unlike Convolutional Neural Networks (CNN), it is known that
the performance of ViT saturates quickly with depth increasing, due to the
observed attention collapse or patch uniformity. Despite a couple of empirical
solutions, a rigorous framework studying on this scalability issue remains
elusive. In this paper, we first establish a rigorous theory framework to
analyze ViT features from the Fourier spectrum domain. We show that the
self-attention mechanism inherently amounts to a low-pass filter, which
indicates when ViT scales up its depth, excessive low-pass filtering will cause
feature maps to only preserve their Direct-Current (DC) component. We then
propose two straightforward yet effective techniques to mitigate the
undesirable low-pass limitation. The first technique, termed AttnScale,
decomposes a self-attention block into low-pass and high-pass components, then
rescales and combines these two filters to produce an all-pass self-attention
matrix. The second technique, termed FeatScale, re-weights feature maps on
separate frequency bands to amplify the high-frequency signals. Both techniques
are efficient and hyperparameter-free, while effectively overcoming relevant
ViT training artifacts such as attention collapse and patch uniformity. By
seamlessly plugging in our techniques to multiple ViT variants, we demonstrate
that they consistently help ViTs benefit from deeper architectures, bringing up
to 1.1% performance gains "for free" (e.g., with little parameter overhead). We
publicly release our codes and pre-trained models at
this https URL

本研究针对 Vision Transformer 深度增加时性能达到饱和的问题提出了两种有效而无需超参数的技术 AttnScale 和 FeatScale，能够有效克服与注意力折叠和补丁均匀性等相关的 ViT 训练伪像。

基于傅里叶域分析的深度视觉变换器中的防过度平滑技术：从理论到实践

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain  Analysis: From Theory to Practice

Graph Convolutional Networks (GCNs) have become a crucial tool on learning
representations of graph vertices. The main challenge of adapting GCNs on
large-scale graphs is the scalability issue that it incurs heavy cost both in
computation and memory due to the uncontrollable neighborhood expansion across
layers. In this paper, we accelerate the training of GCNs through developing an
adaptive layer-wise sampling method. By constructing the network layer by layer
in a top-down passway, we sample the lower layer conditioned on the top one,
where the sampled neighborhoods are shared by different parent nodes and the
over expansion is avoided owing to the fixed-size sampling. More importantly,
the proposed sampler is adaptive and applicable for explicit variance
reduction, which in turn enhances the training of our method. Furthermore, we
propose a novel and economical approach to promote the message passing over
distant nodes by applying skip connections. Intensive experiments on several
benchmarks verify the effectiveness of our method regarding the classification
accuracy while enjoying faster convergence speed.

本研究提出了一种自适应逐层采样方法和跨越较远节点的消息传递方法，以解决图形卷积网络在大规模图中的可扩展性问题，并通过实验验证了其有效性和更快的收敛速度。