Predictor-based methods have substantially enhanced Neural Architecture
Search (NAS) optimization. The efficacy of these predictors is largely
influenced by the method of encoding neural network architectures. While
traditional encodings used an adjacency matrix describing the graph structure
of a neural network, novel encodings embrace a variety of approaches from
unsupervised pretraining of latent representations to vectors of zero-cost
proxies. In this paper, we categorize and investigate neural encodings from
three main types: structural, learned, and score-based. Furthermore, we extend
these encodings and introduce \textit{unified encodings}, that extend NAS
predictors to multiple search spaces. Our analysis draws from experiments
conducted on over 1.5 million neural network architectures on NAS spaces such
as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and
TransNASBench-101. Building on our study, we present our predictor
\textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN
integrates critical insights on predictor design, transfer learning, and
\textit{unified encodings} to enable more than an order of magnitude cost
reduction for training NAS accuracy predictors. Our implementation and
encodings for all neural networks are open-sourced at
\href{https://github.com/abdelfattah-lab/flan_nas}{this https URL}.

利用预测器进行神经结构搜索（NAS）优化的方法大大提高了其有效性，本文对神经网络架构的编码方法进行了分类和研究，包括结构编码、学习编码和基于分数的编码，并引入了统一编码方法，进一步扩展了 NAS 预测器的应用范围。此研究基于在 NASBench-101（NB101）、NB201、NB301、Network Design Spaces（NDS）和 TransNASBench-101 等 NAS 领域超过 150 万个神经网络架构进行的实验证明，我们提出的预测器 FLAN（流动关注）通过融合预测器设计、迁移学习和统一编码，在训练 NAS 准确性预测器时能大幅度降低成本。所有神经网络的实现和编码方式均已在 https://github.com/abdelfattah-lab/flan_nas 开源。

基于预测的神经结构搜索的编码方式

Encodings for Prediction-based Neural Architecture Search

This paper explores a simple method for improving the zero-shot learning
abilities of language models. We show that instruction tuning -- finetuning
language models on a collection of tasks described via instructions --
substantially improves zero-shot performance on unseen tasks.
We take a 137B parameter pretrained language model and instruction-tune it on
over 60 NLP tasks verbalized via natural language instruction templates. We
evaluate this instruction-tuned model, which we call FLAN, on unseen task
types. FLAN substantially improves the performance of its unmodified
counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we
evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE,
BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number
of finetuning datasets, model scale, and natural language instructions are key
to the success of instruction tuning.

通过对自然语言指令模板中的 60 个自然语言处理任务进行调整，我们将一个 137B 预训练语言模型调整为 FLAN 并在未看见的任务数据上进行评估，结果表明，通过指令调整，可以大大改善它在未看见任务上的性能并在 20 个任务上超越了 175B GPT-3 的零样本性能。