Vision transformers (ViTs) inherited the success of NLP but their structures
have not been sufficiently investigated and optimized for visual tasks. One of
the simplest solutions is to directly search the optimal one via the widely
used neural architecture search (NAS) in CNNs. However, we empirically find
this straightforward adaptation would encounter catastrophic failures and be
frustratingly unstable for the training of superformer. In this paper, we argue
that since ViTs mainly operate on token embeddings with little inductive bias,
imbalance of channels for different architectures would worsen the
weight-sharing assumption and cause the training instability as a result.
Therefore, we develop a new cyclic weight-sharing mechanism for token
embeddings of the ViTs, which enables each channel could more evenly contribute
to all candidate architectures. Besides, we also propose identity shifting to
alleviate the many-to-one issue in superformer and leverage weak augmentation
and regularization techniques for more steady training empirically. Based on
these, our proposed method, ViTAS, has achieved significant superiority in both
DeiT- and Twins-based ViTs. For example, with only $1.4$G FLOPs budget, our
searched architecture has $3.3\%$ ImageNet-$1$k accuracy than the baseline
DeiT. With $3.0$G FLOPs, our results achieve $82.0\%$ accuracy on
ImageNet-$1$k, and $45.9\%$ mAP on COCO$2017$ which is $2.4\%$ superior than
other ViTs.

本文提出了 ViTAS 方法，其中使用循环权重共享机制和身份移位来解决目标嵌入的失衡问题，并辅以弱数据增强和规范化技术，以获得稳定的训练结果。实验证明，与其他常用方法相比，ViTAS 取得了极高的性能提升。

ViTAS: 视觉 Transformer 架构搜索

ViTAS: Vision Transformer Architecture Search

Despite recent advances in representation learning in hypercomplex (HC)
space, this subject is still vastly unexplored in the context of graphs.
Motivated by the complex and quaternion algebras, which have been found in
several contexts to enable effective representation learning that inherently
incorporates a weight-sharing mechanism, we develop graph neural networks that
leverage the properties of hypercomplex feature transformation. In particular,
in our proposed class of models, the multiplication rule specifying the algebra
itself is inferred from the data during training. Given a fixed model
architecture, we present empirical evidence that our proposed model
incorporates a regularization effect, alleviating the risk of overfitting. We
also show that for fixed model capacity, our proposed method outperforms its
corresponding real-formulated GNN, providing additional confirmation for the
enhanced expressivity of HC embeddings. Finally, we test our proposed
hypercomplex GNN on several open graph benchmark datasets and show that our
models reach state-of-the-art performance while consuming a much lower memory
footprint with 70& fewer parameters. Our implementations are available at
this https URL

本研究旨在开发利用超复特征变换的图神经网络，通过推断数据自身的乘法规则来实现一种新的模型，其具有正则化效果，能够显著提升 HC 嵌入的表达能力，相比对应的实数 GNN，达到了 more state-of-the-art 的性能，同时内存占用和参数数量都更少。