Differentiable architecture search (DARTS) yields highly efficient
gradient-based neural architecture search (NAS) by relaxing the discrete
operation selection to optimize continuous architecture parameters that maps
NAS from the discrete optimization to a continuous problem. DARTS then remaps
the relaxed supernet back to the discrete space by one-off post-search pruning
to obtain the final architecture (finalnet). Some emerging works argue that
this remap is inherently prone to mismatch the network between training and
evaluation which leads to performance discrepancy and even model collapse in
extreme cases. We propose to close the gap between the relaxed supernet in
training and the pruned finalnet in evaluation through utilizing small
temperature to sparsify the continuous distribution in the training phase. To
this end, we first formulate sparse-noisy softmax to get around gradient
saturation. We then propose an exponential temperature schedule to better
control the outbound distribution and elaborate an entropy-based adaptive
scheme to finally achieve the enhancement. We conduct extensive experiments to
verify the efficiency and efficacy of our method.

通过利用小的温度值来稀疏化训练阶段中的连续分布，来缩小训练中弱化超网络与评估中的修剪后网络之间的差距，以提高 Differentiable Architecture Search 的效率和功效。

微温度即可完成可微架构搜索

Small Temperature is All You Need for Differentiable Architecture Search

Parallel tempering is a generic Markov chain Monte Carlo sampling method
which allows good mixing with multimodal target distributions, where
conventional Metropolis-Hastings algorithms often fail. The mixing properties
of the sampler depend strongly on the choice of tuning parameters, such as the
temperature schedule and the proposal distribution used for local exploration.
We propose an adaptive algorithm which tunes both the temperature schedule and
the parameters of the random-walk Metropolis kernel automatically. We prove the
convergence of the adaptation and a strong law of large numbers for the
algorithm. We illustrate the performance of our method with examples. Our
empirical findings indicate that the algorithm can cope well with different
kind of scenarios without prior tuning.

提出了一种自适应算法，可以自动调节温度进度表和随机游走 Metropolis 核的参数。经验结果表明该算法无需先验调整，可以很好地应对不同的场景。