We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

本文研究了学会学习的问题：推断出适用于从未知分布中抽样的任务的学习算法。作为算法类别，我们考虑正则化真实风险的随机梯度下降。我们提出了一个元算法，增量地更新偏置，从而解决了从一系列任务中估算偏置的问题。我们的结果表明，当任务数量增加且方差相对较小时，我们的学习方法在学习效果上具有重大优势。

用带偏正则化的学习-学习随机梯度下降算法