We propose a novel VQA dataset, based on picture stories designed for
educating young children, that aims to facilitate comprehensive evaluation and
characterization of vision-language models on comprehension tasks. Unlike
current VQA datasets that often focus on fact-based memorization and simple
reasoning tasks without principled scientific grounding, we collect data
containing tasks reflecting different levels of comprehension and underlying
cognitive processes, as laid out in Bloom's Taxonomy, a classic framework
widely adopted in education research. The proposed BloomVQA dataset can be
mapped to a hierarchical graph-based representation of visual stories, enabling
automatic data augmentation and novel measures characterizing model consistency
across the underlying taxonomy. We demonstrate graded evaluation and
reliability analysis based on our proposed consistency metrics on
state-of-the-art vision-language models. Our results suggest that, while
current models achieve the most gain on low-level comprehension tasks, they
generally fall short on high-level tasks requiring more advanced comprehension
and cognitive skills, as 38.0% drop in VQA accuracy is observed comparing
lowest and highest level tasks. Furthermore, current models show consistency
patterns misaligned with human comprehension in various scenarios, suggesting
emergent structures of model behaviors.

我们提出了一个新颖的视觉问答（VQA）数据集，基于为教育幼儿而设计的图片故事，旨在通过理解任务对视觉语言模型进行全面评估和特征化。该数据集包含不同层次的理解任务和底层认知过程，可以映射到基于图形的视觉故事表示，实现自动数据增强和模型一致性特征。我们的结果表明，尽管现有模型在低级别理解任务上取得了最大的进步，但在需要更高级理解和认知技能的高级任务上，它们普遍表现不佳，最高级别任务的 VQA 准确度下降了 38.0%。此外，目前的模型在各种情境中与人类理解的一致性模式不匹配，表明模型行为的新出现结构。

BloomVQA：评估层次化多模态理解

BloomVQA: Assessing Hierarchical Multi-modal Comprehension

Large language models are capable of displaying a wide range of abilities
that are not directly connected with the task for which they are trained:
predicting the next words of human-written texts. In this article, I discuss
the nature of this indirect acquisition process and its relation to other known
indirect processes. I argue that an important side effect of such indirect
acquisition is the development of integrated abilities. I discuss the extent to
which the abilities developed by large language models are predictable.
Finally, I briefly discuss the relation between the cognitive skills acquired
by these systems and human cognition.

大型语言模型的能力不仅局限于预测人类书写文本的下一句，还包括间接获取能力、综合能力的发展、可预测性以及与人类认知的关系。

关于大型语言模型的意外能力

On the Unexpected Abilities of Large Language Models

A robot operating in unstructured environments must be able to discriminate
between different grasping styles depending on the prospective manipulation
task. Having a system that allows learning from remote non-expert
demonstrations can very feasibly extend the cognitive skills of a robot for
task-oriented grasping. We propose a novel two-step framework towards this aim.
The first step involves grasp area estimation by segmentation. We receive grasp
area demonstrations for a new task via interactive segmentation, and learn from
these few demonstrations to estimate the required grasp area on an unseen scene
for the given task. The second step is autonomous grasp estimation in the
segmented region. To train the segmentation network for few-shot learning, we
built a grasp area segmentation (GAS) dataset with 10089 images grouped into
1121 segmentation tasks. We benefit from an efficient meta learning algorithm
for training for few-shot adaptation. Experimental evaluation showed that our
method successfully detects the correct grasp area on the respective objects in
unseen test scenes and effectively allows remote teaching of new grasp
strategies by non-experts.

本论文提出了一个新的两步框架，使机器人在非结构化环境中学会抓取并能够从远程的非专家演示中学习，通过分割的方式进行操作，在进行少量示范的情况下估计给定任务的未见场景所需的抓取区域，并有效地允许非专家进行新的抓取策略的远程教学。