Vision Transformers (ViTs) have underpinned the recent breakthroughs in
computer vision. However, designing the architectures of ViTs is laborious and
heavily relies on expert knowledge. To automate the design process and
incorporate deployment flexibility, one-shot neural architecture search
decouples the supernet training and architecture specialization for diverse
deployment scenarios. To cope with an enormous number of sub-networks in the
supernet, existing methods treat all architectures equally important and
randomly sample some of them in each update step during training. During
architecture search, these methods focus on finding architectures on the Pareto
frontier of performance and resource consumption, which forms a gap between
training and deployment. In this paper, we devise a simple yet effective
method, called FocusFormer, to bridge such a gap. To this end, we propose to
learn an architecture sampler to assign higher sampling probabilities to those
architectures on the Pareto frontier under different resource constraints
during supernet training, making them sufficiently optimized and hence
improving their performance. During specialization, we can directly use the
well-trained architecture sampler to obtain accurate architectures satisfying
the given resource constraint, which significantly improves the search
efficiency. Extensive experiments on CIFAR-100 and ImageNet show that our
FocusFormer is able to improve the performance of the searched architectures
while significantly reducing the search cost. For example, on ImageNet, our
FocusFormer-Ti with 1.4G FLOPs outperforms AutoFormer-Ti by 0.5% in terms of
the Top-1 accuracy.

Vision Transformers 是目前计算机视觉领域的重要技术，这篇论文提出了一种名为 FocusFormer 的方法，它通过学习一个架构采样器，在不同资源约束条件下分配更高的采样概率，从而提高架构的性能，并显著降低搜索成本。

通过架构采样器专注于我们需要的内容

FocusFormer: Focusing on What We Need via Architecture Sampler

Realistic use of neural networks often requires adhering to multiple
constraints on latency, energy and memory among others. A popular approach to
find fitting networks is through constrained Neural Architecture Search (NAS),
however, previous methods enforce the constraint only softly. Therefore, the
resulting networks do not exactly adhere to the resource constraint and their
accuracy is harmed. In this work we resolve this by introducing Hard
Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate
formulation of the expected resource requirement and a scalable search method
that satisfies the hard constraint throughout the search. Our experiments show
that HardCoRe-NAS generates state-of-the-art architectures, surpassing other
NAS methods, while strictly satisfying the hard resource constraints without
any tuning required.

通过引入 HardCoRe-NAS 方法，我们精确地满足了各种资源约束，并在不需要任何调整的情况下生成了具有 state-of-the-art 性能的架构。