Weight sharing is a popular approach to reduce the cost of neural
architecture search (NAS) by reusing the weights of shared operators from
previously trained child models. However, the rank correlation between the
estimated accuracy and ground truth accuracy of those child models is low due
to the interference among different child models caused by weight sharing. In
this paper, we investigate the interference issue by sampling different child
models and calculating the gradient similarity of shared operators, and
observe: 1) the interference on a shared operator between two child models is
positively correlated with the number of different operators; 2) the
interference is smaller when the inputs and outputs of the shared operator are
more similar. Inspired by these two observations, we propose two approaches to
mitigate the interference: 1) MAGIC-T: rather than randomly sampling child
models for optimization, we propose a gradual modification scheme by modifying
one operator between adjacent optimization steps to minimize the interference
on the shared operators; 2) MAGIC-A: forcing the inputs and outputs of the
operator across all child models to be similar to reduce the interference.
Experiments on a BERT search space verify that mitigating interference via each
of our proposed methods improves the rank correlation of super-pet and
combining both methods can achieve better results. Our discovered architecture
outperforms RoBERTa$_{\rm base}$ by 1.1 and 0.6 points and ELECTRA$_{\rm base}$
by 1.6 and 1.1 points on the dev and test set of GLUE benchmark. Extensive
results on the BERT compression, reading comprehension and ImageNet task
demonstrate the effectiveness and generality of our proposed methods.

本文探讨神经体系结构搜索中的权重共享问题并提出了 MAGIC-T 和 MAGIC-A 两种方法来缓解共享运算符之间的干扰。实验证明，我们的方法能够提高超级神经网络的排序相关性，并在各种任务中验证了其有效性和普适性。