We consider learning a convex combination of basis models, and present some
new theoretical and empirical results that demonstrate the effectiveness of a
greedy approach. Theoretically, we first consider whether we can use linear,
instead of convex, combinations, and obtain generalization results similar to
existing ones for learning from a convex hull. We obtain a negative result that
even the linear hull of very simple basis functions can have unbounded
capacity, and is thus prone to overfitting; on the other hand, convex hulls are
still rich but have bounded capacities. Secondly, we obtain a generalization
bound for a general class of Lipschitz loss functions. Empirically, we first
discuss how a convex combination can be greedily learned with early stopping,
and how a convex combination can be non-greedily learned when the number of
basis models is known a priori. Our experiments suggest that the greedy scheme
is competitive with or better than several baselines, including boosting and
random forests. The greedy algorithm requires little effort in hyper-parameter
tuning, and also seems able to adapt to the underlying complexity of the
problem. Our code is available at this https URL

通过贪心算法学习基础模型的凸组合，得出结论：凸壳具有有限容量，良好的泛化性能，而线性壳的容量无限，容易出现过拟合情况。基于此，使用凸组合并结合早停法和拟定计算量可得的方法，可以实现贪心和非贪心学习。实验表明，贪心学习是一种有效的方法，且适应性强。