Training machine learning models on massive datasets incurs substantial
computational costs. To alleviate such costs, there has been a sustained effort
to develop data-efficient training methods that can carefully select subsets of
the training examples that generalize on par with the full training data.
However, existing methods are limited in providing theoretical guarantees for
the quality of the models trained on the extracted subsets, and may perform
poorly in practice. We propose AdaCore, a method that leverages the geometry of
the data to extract subsets of the training examples for efficient machine
learning. The key idea behind our method is to dynamically approximate the
curvature of the loss function via an exponentially-averaged estimate of the
Hessian to select weighted subsets (coresets) that provide a close
approximation of the full gradient preconditioned with the Hessian. We prove
rigorous guarantees for the convergence of various first and second-order
methods applied to the subsets chosen by AdaCore. Our extensive experiments
show that AdaCore extracts coresets with higher quality compared to baselines
and speeds up training of convex and non-convex machine learning models, such
as logistic regression and neural networks, by over 2.9x over the full data and
4.5x over random subsets.

提出 AdaCore 方法，该方法利用数据的几何形状，用指数平均估计的 Hessian 近似曲率，选择加权的子集，提供与使用完整数据的梯度预处理的负梯度近似，实现高效的机器学习训练，并对其给出理论支持和证明，相比基线提取的 coresets 质量更高，速度快 2.9 倍以上。