BriefGPT.xyz
Apr, 2021
神经网络损失函数中单调线性插值的分析
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
HTML
PDF
James Lucas, Juhan Bae, Michael R. Zhang, Stanislav Fort, Richard Zemel...
TL;DR
通过分析神经网络的初始参数和训练后的参数之间的线性插值关系,本文首次给出了神经网络训练中的单调线性插值(MLI)现象,通过利用微分几何理论,提供了MSE中MLI出现的充分条件,并发现了当网络权重远离初始化时,将出现违反MLI现象的情况。
Abstract
Linear interpolation between initial neural network parameters and converged parameters after training with
stochastic gradient descent
(SGD) typically leads to a monotonic decrease in the training objective. This
monot
→