A wide range of empirical and theoretical works have shown that
overparameterisation can amplify the performance of neural networks. According
to the lottery ticket hypothesis, overparameterised networks have an increased
chance of containing a sub-network that is well-initialised to solve the task
at hand. A more parsimonious approach, inspired by animal learning, consists in
guiding the learner towards solving the task by curating the order of the
examples, i.e. providing a curriculum. However, this learning strategy seems to
be hardly beneficial in deep learning applications. In this work, we undertake
an analytical study that connects curriculum learning and overparameterisation.
In particular, we investigate their interplay in the online learning setting
for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results
show that a high degree of overparameterisation -- while simplifying the
problem -- can limit the benefit from curricula, providing a theoretical
account of the ineffectiveness of curricula in deep learning.

通过分析，我们研究了课程学习和超参数化在在线学习环境中的相互作用，结果显示，过度参数化可以简化问题，但可能限制了课程学习的效果，从而为深度学习中课程学习的无效性提供了理论解释。

在神经网络中倾斜彩票的机会：过参数化和课程表的相互作用

Tilting the Odds at the Lottery: the Interplay of Overparameterisation  and Curricula in Neural Networks

Compact neural network offers many benefits for real-world applications.
However, it is usually challenging to train the compact neural networks with
small parameter sizes and low computational costs to achieve the same or better
model performance compared to more complex and powerful architecture. This is
particularly true for multitask learning, with different tasks competing for
resources. We present a simple, efficient and effective multitask learning
overparameterisation neural network design by overparameterising the model
architecture in training and sharing the overparameterised model parameters
more effectively across tasks, for better optimisation and generalisation.
Experiments on two challenging multitask datasets (NYUv2 and COCO) demonstrate
the effectiveness of the proposed method across various convolutional networks
and parameter sizes.

紧凑型神经网络面向多任务学习的过参数化设计以提高优化和泛化性能，通过在训练过程中对模型架构进行过参数化，并更有效地共享过参数化的模型参数，以在小参数大小和低计算成本下获得与更复杂和强大架构相同或更好的模型性能。在两个具有挑战性的多任务数据集（NYUv2 和 COCO）上的实验证明了所提方法在各种卷积网络和参数大小下的有效性。