Iterative procedures for parameter estimation based on stochastic gradient
descent allow the estimation to scale to massive data sets. However, in both
theory and practice, they suffer from numerical instability. Moreover, they are
statistically inefficient as estimators of the true parameter value. To address
these two issues, we propose a new iterative procedure termed averaged implicit
SGD (AI-SGD). For statistical efficiency, AI-SGD employs averaging of the
iterates, which achieves the optimal Cram\'{e}r-Rao bound under strong
convexity, i.e., it is an optimal unbiased estimator of the true parameter
value. For numerical stability, AI-SGD employs an implicit update at each
iteration, which is related to proximal operators in optimization. In practice,
AI-SGD achieves competitive performance with other state-of-the-art procedures.
Furthermore, it is more stable than averaging procedures that do not employ
proximal updates, and is simple to implement as it requires fewer tunable
hyperparameters than procedures that do employ proximal updates.

提出一种基于平均隐式（averaged implicit）随机梯度下降的迭代过程，旨在解决参数估计过程中的数值不稳定性和统计效率问题。实践证明这种方法比其他现有方法表现更为出色。