In recent years, meta-learning, in which a model is trained on a family of
tasks (i.e. a task distribution), has emerged as an approach to training neural
networks to perform tasks that were previously assumed to require structured
representations, making strides toward closing the gap between humans and
machines. However, we argue that evaluating meta-learning remains a challenge,
and can miss whether meta-learning actually uses the structure embedded within
the tasks. These meta-learners might therefore still be significantly different
from humans learners. To demonstrate this difference, we first define a new
meta-reinforcement learning task in which a structured task distribution is
generated using a compositional grammar. We then introduce a novel approach to
constructing a "null task distribution" with the same statistical complexity as
this structured task distribution but without the explicit rule-based structure
used to generate the structured task. We train a standard meta-learning agent,
a recurrent network trained with model-free reinforcement learning, and compare
it with human performance across the two task distributions. We find a double
dissociation in which humans do better in the structured task distribution
whereas agents do better in the null task distribution -- despite comparable
statistical complexity. This work highlights that multiple strategies can
achieve reasonable meta-test performance, and that careful construction of
control task distributions is a valuable way to understand which strategies
meta-learners acquire, and how they might differ from humans.

通过构建基于组合语法的结构化任务分布和相同统计复杂度但没有明确的基于规则的结构的 “空任务分布”，比较人类表现与标准元学习代理在两个任务分布中的表现，发现虽然统计复杂度相当，但人类在结构化任务分布中表现更好，而代理在空任务分布中表现更好，这表明控制任务分布的谨慎构建是理解元学习者获得的策略及其与人类的差异之一种有价值的方法。