Recent one-shot Neural Architecture Search algorithms rely on training a
hardware-agnostic super-network tailored to a specific task and then extracting
efficient sub-networks for different hardware platforms. Popular approaches
separate the training of super-networks from the search for sub-networks, often
employing predictors to alleviate the computational overhead associated with
search. Additionally, certain methods also incorporate the quantization policy
within the search space. However, while the quantization policy search for
convolutional neural networks is well studied, the extension of these methods
to transformers and especially foundation models remains under-explored. In
this paper, we demonstrate that by using multi-objective search algorithms
paired with lightly trained predictors, we can efficiently search for both the
sub-network architecture and the corresponding quantization policy and
outperform their respective baselines across different performance objectives
such as accuracy, model size, and latency. Specifically, we demonstrate that
our approach performs well across both uni-modal (ViT and BERT) and multi-modal
(BEiT-3) transformer-based architectures as well as convolutional architectures
(ResNet). For certain networks, we demonstrate an improvement of up to $4.80x$
and $3.44x$ for latency and model size respectively, without degradation in
accuracy compared to the fully quantized INT8 baselines.

通过使用多目标搜索算法与轻度训练的预测器相结合，我们展示了在准确性、模型大小和延迟等不同性能目标中，我们的方法在 ViT、BERT、BEiT-3 和 ResNet 等转换器和卷积网络的架构上比各自的基准表现更好，特定网络的性能提升在延迟方面最高达到 4.80 倍，在模型大小方面最高达到 3.44 倍，与全量化 INT8 基准相比准确性不降低。

SimQ-NAS: 同時量化策略與神經架構搜尋

SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture  Search

One-Shot Neural Architecture Search (NAS) algorithms often rely on training a
hardware agnostic super-network for a domain specific task. Optimal
sub-networks are then extracted from the trained super-network for different
hardware platforms. However, training super-networks from scratch can be
extremely time consuming and compute intensive especially for large models that
rely on a two-stage training process of pre-training and fine-tuning. State of
the art pre-trained models are available for a wide range of tasks, but their
large sizes significantly limits their applicability on various hardware
platforms. We propose InstaTune, a method that leverages off-the-shelf
pre-trained weights for large models and generates a super-network during the
fine-tuning stage. InstaTune has multiple benefits. Firstly, since the process
happens during fine-tuning, it minimizes the overall time and compute resources
required for NAS. Secondly, the sub-networks extracted are optimized for the
target task, unlike prior work that optimizes on the pre-training objective.
Finally, InstaTune is easy to "plug and play" in existing frameworks. By using
multi-objective evolutionary search algorithms along with lightly trained
predictors, we find Pareto-optimal sub-networks that outperform their
respective baselines across different performance objectives such as accuracy
and MACs. Specifically, we demonstrate that our approach performs well across
both unimodal (ViT and BERT) and multi-modal (BEiT-3) transformer based
architectures.

提出了 InstaTune 方法，该方法利用现成的预训练权重为大型模型生成超级网络，通过使用多目标进化搜索算法和轻度训练的预测器，得到在准确度和 MACs 等性能目标方面优于基线的帕累托最优子网络。

InstaTune：微调期间的即时神经架构搜索

InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning

Recent advances in Neural Architecture Search (NAS) which extract specialized
hardware-aware configurations (a.k.a. "sub-networks") from a hardware-agnostic
"super-network" have become increasingly popular. While considerable effort has
been employed towards improving the first stage, namely, the training of the
super-network, the search for derivative high-performing sub-networks is still
largely under-explored. For example, some recent network morphism techniques
allow a super-network to be trained once and then have hardware-specific
networks extracted from it as needed. These methods decouple the super-network
training from the sub-network search and thus decrease the computational burden
of specializing to different hardware platforms. We propose a comprehensive
system that automatically and efficiently finds sub-networks from a pre-trained
super-network that are optimized to different performance metrics and hardware
configurations. By combining novel search tactics and algorithms with
intelligent use of predictors, we significantly decrease the time needed to
find optimal sub-networks from a given super-network. Further, our approach
does not require the super-network to be refined for the target task a priori,
thus allowing it to interface with any super-network. We demonstrate through
extensive experiments that our system works seamlessly with existing
state-of-the-art super-network training methods in multiple domains. Moreover,
we show how novel search tactics paired with evolutionary algorithms can
accelerate the search process for ResNet50, MobileNetV3 and Transformer while
maintaining objective space Pareto front diversity and demonstrate an 8x faster
search result than the state-of-the-art Bayesian optimization WeakNAS approach.

本文提出了一个自动且高效地从预训练的超网络中找到针对不同性能度量和硬件配置进行优化的子网络的综合系统，在多个领域中与现有最先进的超网络训练方法无缝配合，展示了如何将新颖的搜索策略与演化算法相结合，加速 ResNet50、MobileNetV3 和 Transformer 的搜索过程，并展示了比最先进的贝叶斯优化 WeakNAS 方法快 8 倍的搜索结果。