Deep learning has shown remarkable success in the field of clustering
recently. However, how to transfer a trained clustering model on a source
domain to a target domain by leveraging the acquired knowledge to guide the
clustering process remains challenging. Existing deep clustering methods often
lack generalizability to new domains because they typically learn a group of
fixed cluster centroids, which may not be optimal for the new domain
distributions. In this paper, we propose a novel transferable deep clustering
model that can automatically adapt the cluster centroids according to the
distribution of data samples. Rather than learning a fixed set of centroids,
our approach introduces a novel attention-based module that can adapt the
centroids by measuring their relationship with samples. In addition, we
theoretically show that our model is strictly more powerful than some classical
clustering algorithms such as k-means or Gaussian Mixture Model (GMM).
Experimental results on both synthetic and real-world datasets demonstrate the
effectiveness and efficiency of our proposed transfer learning framework, which
significantly improves the performance on target domain and reduces the
computational cost.

通过引入一种新的注意力模型，我们提出了一种可以自动调整聚类中心的可转移的深度聚类模型，该模型在数据样本分布方面自动适应聚类中心，并理论上证明了我们的模型比一些传统聚类算法如 k-means 或高斯混合模型 (GMM) 更强大。在合成和真实数据集上的实验结果证明了我们提出的传输学习框架的有效性和效率，该框架在目标领域的性能表现显著提高并减少了计算成本。

可迁移的深度聚类模型

Transferable Deep Clustering Model

Transformers have shown great potential in computer vision tasks. A common
belief is their attention-based token mixer module contributes most to their
competence. However, recent works show the attention-based module in
Transformers can be replaced by spatial MLPs and the resulted models still
perform quite well. Based on this observation, we hypothesize that the general
architecture of the Transformers, instead of the specific token mixer module,
is more essential to the model's performance. To verify this, we deliberately
replace the attention module in Transformers with an embarrassingly simple
spatial pooling operator to conduct only basic token mixing. Surprisingly, we
observe that the derived model, termed as PoolFormer, achieves competitive
performance on multiple computer vision tasks. For example, on ImageNet-1K,
PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned Vision
Transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with
35%/52% fewer parameters and 50%/62% fewer MACs. The effectiveness of
PoolFormer verifies our hypothesis and urges us to initiate the concept of
"MetaFormer", a general architecture abstracted from Transformers without
specifying the token mixer. Based on the extensive experiments, we argue that
MetaFormer is the key player in achieving superior results for recent
Transformer and MLP-like models on vision tasks. This work calls for more
future research dedicated to improving MetaFormer instead of focusing on the
token mixer modules. Additionally, our proposed PoolFormer could serve as a
starting baseline for future MetaFormer architecture design. Code is available
at this https URL

通过取代 Transformers 中的注意力机制，使用空间 MLPs，从而验证了 Transformers 的总体架构（MetaFormer）对模型性能的贡献更为关键。提出了 MetaFormer 的概念，这是一个抽象出 Transformers 中 token mixer 以外的通用架构，并将提出的 PoolFormer 作为未来 MetaFormer 架构设计的起点基线模型。

MetaFormer 对于计算机视觉是必不可少的

MetaFormer Is Actually What You Need for Vision

Given a sequence of sets, where each set contains an arbitrary number of
elements, the problem of temporal sets prediction aims to predict the elements
in the subsequent set. In practice, temporal sets prediction is much more
complex than predictive modelling of temporal events and time series, and is
still an open problem. Many possible existing methods, if adapted for the
problem of temporal sets prediction, usually follow a two-step strategy by
first projecting temporal sets into latent representations and then learning a
predictive model with the latent representations. The two-step approach often
leads to information loss and unsatisfactory prediction performance. In this
paper, we propose an integrated solution based on the deep neural networks for
temporal sets prediction. A unique perspective of our approach is to learn
element relationship by constructing set-level co-occurrence graph and then
perform graph convolutions on the dynamic relationship graphs. Moreover, we
design an attention-based module to adaptively learn the temporal dependency of
elements and sets. Finally, we provide a gated updating mechanism to find the
hidden shared patterns in different sequences and fuse both static and dynamic
information to improve the prediction performance. Experiments on real-world
data sets demonstrate that our approach can achieve competitive performances
even with a portion of the training data and can outperform existing methods
with a significant margin.

本研究提出了基于深度神经网络的时间序列预测整体解决方案，其中独特的视角是通过构建集合级别的共现图来学习元素关系，并在动态关系图上执行图卷积。此外，设计了基于注意力机制的模块来自适应地学习元素和集合之间的时间依赖性，并提供门控机制来发现不同序列中的隐藏共享模式并融合静态和动态信息来提高预测性能。实验结果表明，即使只有一部分训练数据，我们的方法也可以取得竞争性的表现，并且可以优于现有方法。