For specialized domains, there is often not a wealth of data with which to train large machine learning models. In such limited data / compute settings, various methods exist aiming to $\textit{do more with less}$, such as finetuning from a pretrained model, modulating difficulty levels as data are presented to a model (curriculum learning), and considering the role of model type / size. Approaches to efficient $\textit{machine}$ learning also take inspiration from $\textit{human}$ learning by considering use cases where machine learning systems have access to approximately the same number of words experienced by a 13 year old child (100M words). We investigate the role of 3 primary variables in a limited data regime as part of the multimodal track of the BabyLM challenge. We contrast: (i) curriculum learning, (ii), pretraining (with text-only data), (iii) model type. We modulate these variables and assess them on two types of tasks: (a) multimodal (text+image), and (b) unimodal (text-only) tasks. We find that curriculum learning benefits multimodal evaluations over non-curriclum learning models, particularly when combining text-only pretraining. On text-only tasks, curriculum learning appears to help models with smaller trainable parameter counts. We suggest possible reasons based on architectural differences and training designs as to why one might observe such results.

本研究针对专门领域中数据稀缺的问题，提出了一种通过课程学习优化视觉-语言任务的方法，来提高小规模多模态训练的效果。核心发现显示，相较于非课程学习模型，在结合文本预训练时，课程学习在多模态评估中展现了显著的优势，尤其是在文本单一任务中，小参数模型的训练也受益于课程学习。这一发现为如何在数据有限的环境中有效利用机器学习提供了新的见解。

探索课程学习在视觉-语言任务中的应用：关于小规模多模态训练的研究