Predictor-based methods have substantially enhanced Neural Architecture
Search (NAS) optimization. The efficacy of these predictors is largely
influenced by the method of encoding neural network architectures. While
traditional encodings used an adjacency matrix describing the graph structure
of a neural network, novel encodings embrace a variety of approaches from
unsupervised pretraining of latent representations to vectors of zero-cost
proxies. In this paper, we categorize and investigate neural encodings from
three main types: structural, learned, and score-based. Furthermore, we extend
these encodings and introduce \textit{unified encodings}, that extend NAS
predictors to multiple search spaces. Our analysis draws from experiments
conducted on over 1.5 million neural network architectures on NAS spaces such
as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and
TransNASBench-101. Building on our study, we present our predictor
\textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN
integrates critical insights on predictor design, transfer learning, and
\textit{unified encodings} to enable more than an order of magnitude cost
reduction for training NAS accuracy predictors. Our implementation and
encodings for all neural networks are open-sourced at
\href{https://github.com/abdelfattah-lab/flan_nas}{this https URL}.

利用预测器进行神经结构搜索（NAS）优化的方法大大提高了其有效性，本文对神经网络架构的编码方法进行了分类和研究，包括结构编码、学习编码和基于分数的编码，并引入了统一编码方法，进一步扩展了 NAS 预测器的应用范围。此研究基于在 NASBench-101（NB101）、NB201、NB301、Network Design Spaces（NDS）和 TransNASBench-101 等 NAS 领域超过 150 万个神经网络架构进行的实验证明，我们提出的预测器 FLAN（流动关注）通过融合预测器设计、迁移学习和统一编码，在训练 NAS 准确性预测器时能大幅度降低成本。所有神经网络的实现和编码方式均已在 https://github.com/abdelfattah-lab/flan_nas 开源。

基于预测的神经结构搜索的编码方式

Encodings for Prediction-based Neural Architecture Search

Various hand-designed CNN architectures have been developed, such as VGG,
ResNet, DenseNet, etc., and achieve State-of-the-Art (SoTA) levels on different
tasks. Neural Architecture Search (NAS) now focuses on automatically finding
the best CNN architecture to handle the above tasks. However, the verification
of a searched architecture is very time-consuming and makes predictor-based
methods become an essential and important branch of NAS. Two commonly used
techniques to build predictors are graph-convolution networks (GCN) and
multilayer perceptron (MLP). In this paper, we consider the difference between
GCN and MLP on adjacent operation trails and then propose the Redirected
Adjacent Trails NAS (RATs-NAS) to quickly search for the desired neural network
architecture. The RATs-NAS consists of two components: the Redirected Adjacent
Trails GCN (RATs-GCN) and the Predictor-based Search Space Sampling (P3S)
module. RATs-GCN can change trails and their strengths to search for a better
neural network architecture. DSS can rapidly focus on tighter intervals of
FLOPs in the search space. Based on our observations on cell-based NAS, we
believe that architectures with similar FLOPs will perform similarly. Finally,
the RATs-NAS consisting of RATs-GCN and DSS beats WeakNAS, Arch-Graph, and
others by a significant margin on three sub-datasets of NASBench-201.

本文介绍了图卷积网络（GCN）和多层感知机（MLP）上邻接操作轨迹之间的差异，并提出了 RATs-NAS 来快速搜索所需的神经网络架构。RATs-NAS 由 Redirected Adjacent Trails GCN（RATs-GCN）和 Predictor-based Search Space Sampling （P3S）模块组成。基于本文的观察，认为具有相似的 FLOPs 架构将表现出类似的表现。最后，RATs-NAS 在 NASBench-201 的三个子数据集上显著击败了 WeakNAS，Arch-Graph 和其他竞争对手。