Recently, the weight-sharing technique has significantly speeded up the
training and evaluation procedure of neural architecture search. However, most
existing weight-sharing strategies are solely based on experience or
observation, which makes the searching results lack interpretability and
rationality. In addition, due to the negligence of fairness, current methods
are prone to make misjudgments in module evaluation. To address these problems,
we propose a novel neural architecture search algorithm based on dynamical
isometry. We use the fix point analysis method in the mean field theory to
analyze the dynamics behavior in the steady state random neural network, and
how dynamic isometry guarantees the fairness of weight-sharing based NAS.
Meanwhile, we prove that our module selection strategy is rigorous fair by
estimating the generalization error of all modules with well-conditioned
Jacobian. Extensive experiments show that, with the same size, the architecture
searched by the proposed method can achieve state-of-the-art top-1 validation
accuracy on ImageNet classification. In addition, we demonstrate that our
method is able to achieve better and more stable training performance without
loss of generality.

本文提出了一种基于动力等距原理的新型神经体系结构搜索算法，解决了现有权重共享策略的不足之处，实现了公平性和稳定的训练性能，并在 ImageNet 分类任务中取得了最先进的 top-1 验证准确性。

基于动态等度性的严格公平神经结构搜索

Dynamical Isometry based Rigorous Fair Neural Architecture Search

Modularity is a compelling solution to continual learning (CL), the problem
of modeling sequences of related tasks. Learning and then composing modules to
solve different tasks provides an abstraction to address the principal
challenges of CL including catastrophic forgetting, backward and forward
transfer across tasks, and sub-linear model growth. We introduce local module
composition (LMC), an approach to modular CL where each module is provided a
local structural component that estimates a module's relevance to the input.
Dynamic module composition is performed layer-wise based on local relevance
scores. We demonstrate that agnosticity to task identities (IDs) arises from
(local) structural learning that is module-specific as opposed to the task-
and/or model-specific as in previous works, making LMC applicable to more CL
settings compared to previous works. In addition, LMC also tracks statistics
about the input distribution and adds new modules when outlier samples are
detected. In the first set of experiments, LMC performs favorably compared to
existing methods on the recent Continual Transfer-learning Benchmark without
requiring task identities. In another study, we show that the locality of
structural learning allows LMC to interpolate to related but unseen tasks
(OOD), as well as to compose modular networks trained independently on
different task sequences into a third modular network without any fine-tuning.
Finally, in search for limitations of LMC we study it on more challenging
sequences of 30 and 100 tasks, demonstrating that local module selection
becomes much more challenging in presence of a large number of candidate
modules. In this setting best performing LMC spawns much fewer modules compared
to an oracle based baseline, however, it reaches a lower overall accuracy. The
codebase is available under this https URL

通过探索局部模块组合（LMC）的方法，使用结构性的学习和模块选择，LMC 可以为连续学习提供较好的解决方案。