How (dis)similar are the learning trajectories of vision-language models and
children? Recent modeling work has attempted to understand the gap between
models' and humans' data efficiency by constructing models trained on less
data, especially multimodal naturalistic data. However, such models are often
evaluated on adult-level benchmarks, with limited breadth in language abilities
tested, and without direct comparison to behavioral data. We introduce
DevBench, a multimodal benchmark comprising seven language evaluation tasks
spanning the domains of lexical, syntactic, and semantic ability, with
behavioral data from both children and adults. We evaluate a set of
vision-language models on these tasks, comparing models and humans not only on
accuracy but on their response patterns. Across tasks, models exhibit variation
in their closeness to human response patterns, and models that perform better
on a task also more closely resemble human behavioral responses. We also
examine the developmental trajectory of OpenCLIP over training, finding that
greater training results in closer approximations to adult response patterns.
DevBench thus provides a benchmark for comparing models to human language
development. These comparisons highlight ways in which model and human language
learning processes diverge, providing insight into entry points for improving
language models.

通过构建模型以较少的、多模态自然数据进行训练，并将其与行为数据进行直接比较，我们介绍了 DevBench，一个包括七个跨越词汇、句法和语义能力领域的语言评估任务的多模态基准。在这些任务中，模型在准确性和回应模式上与人类表现出差异。通过比较模型与人类在任务上的表现，我们发现在语言发展过程中模型与人类的差异，并提供了改进语言模型的切入点。

DevBench：一个用于语言学习的多模态发展基准

DevBench: A multimodal developmental benchmark for language learning

This work aims to automatically evaluate whether the language development of
children is age-appropriate. Validated speech and language tests are used for
this purpose to test the auditory memory. In this work, the task is to
determine whether spoken nonwords have been uttered correctly. We compare
different approaches that are motivated to model specific language structures:
Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated
embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR
acoustic model). Each of the approaches provides input for VGG-like 5-layer CNN
classifiers. We also examine the adaptation per nonword. The evaluation of the
proposed systems was performed using recordings from different kindergartens of
spoken nonwords. ECAPA-TDNN and low-level FFT features do not explicitly model
phonetic information; wav2vec2.0 is trained on grapheme labels, our ASR
acoustic model features contain (sub-)phonetic information. We found that the
more granular the phonetic modeling is, the higher are the achieved recognition
rates. The best system trained on ASR acoustic model features with VTLN
achieved an accuracy of 89.4% and an area under the ROC (Receiver Operating
Characteristic) curve (AUC) of 0.923. This corresponds to an improvement in
accuracy of 20.2% and AUC of 0.309 relative compared to the FFT-baseline.

本研究旨在通过使用不同方法进行语音分析，包括低级特征、说话者嵌入、音素和 ASR 音频模型，使用 VGG - 类 5 层 CNN 分类器，以便自动评估儿童的语言发展是否年龄适宜，证明如果精细调节音素模型，精度可以获得显著提高，并且 ASR 伴音模型提取的语音特征在此任务中表现最佳。

学前儿童语言发展测试中的伪词发音分类

Nonwords Pronunciation Classification in Language Development Tests for Preschool Children

During their first years of life, infants learn the language(s) of their
environment at an amazing speed despite large cross cultural variations in
amount and complexity of the available language input. Understanding this
simple fact still escapes current cognitive and linguistic theories. Recently,
spectacular progress in the engineering science, notably, machine learning and
wearable technology, offer the promise of revolutionizing the study of
cognitive development. Machine learning offers powerful learning algorithms
that can achieve human-like performance on many linguistic tasks. Wearable
sensors can capture vast amounts of data, which enable the reconstruction of
the sensory experience of infants in their natural environment. The project of
'reverse engineering' language development, i.e., of building an effective
system that mimics infant's achievements appears therefore to be within reach.
Here, we analyze the conditions under which such a project can contribute to
our scientific understanding of early language development. We argue that
instead of defining a sub-problem or simplifying the data, computational models
should address the full complexity of the learning situation, and take as input
the raw sensory signals available to infants. This implies that (1) accessible
but privacy-preserving repositories of home data be setup and widely shared,
and (2) models be evaluated at different linguistic levels through a benchmark
of psycholinguist tests that can be passed by machines and humans alike, (3)
linguistically and psychologically plausible learning architectures be scaled
up to real data using probabilistic/optimization principles from machine
learning. We discuss the feasibility of this approach and present preliminary
results.

通过机器学习和可穿戴技术等工程科学的显著进展，研究如何在复杂的学习环境下，通过分析婴儿的语言发展过程、比较婴儿与机器的学习成果以及通过多种测试手段对机器学习进行评估等一系列条件和措施来揭示婴儿语言习得的科学原理。