BriefGPT.xyz
Jul, 2021
KAISA:用于深度神经网络的自适应二阶优化框架
KAISA: An Adaptive Second-order Optimizer Framework for Deep Neural Networks
HTML
PDF
J. Gregory Pauloski, Qi Huang, Lei Huang, Shivaram Venkataraman, Kyle Chard...
TL;DR
KAISA是一种适用于大型神经网络的可适应、改进和可扩展的基于K-FAC的二阶优化器框架,它在比较于原始优化器相同的全局批处理大小下,收敛速度快18.1%-36.3%。
Abstract
Kronecker-factored Approximate Curvature (
k-fac
) has recently been shown to converge faster in
deep neural network
(DNN) training than stochastic gradient descent (SGD); however,
→