The capability of intelligent models to extrapolate and comprehend changes in
object states is a crucial yet demanding aspect of AI research, particularly
through the lens of human interaction in real-world settings. This task
involves describing complex visual environments, identifying active objects,
and interpreting their changes as conveyed through language. Traditional
methods, which isolate object captioning and state change detection, offer a
limited view of dynamic environments. Moreover, relying on a small set of
symbolic words to represent changes has restricted the expressiveness of
language. To address these challenges, in this paper, we introduce the Object
State Captioning and State Change Representation (OSCaR) dataset and benchmark.
OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique
objects from various egocentric video collections. It sets a new testbed for
evaluating multimodal large language models (MLLMs). Our experiments
demonstrate that while MLLMs show some skill, they lack a full understanding of
object state changes. The benchmark includes a fine-tuned model that, despite
initial capabilities, requires significant improvements in accuracy and
generalization ability for effective understanding of these changes. Our code
and dataset are available at this https URL

通过介绍 Object State Captioning and State Change Representation (OSCaR) 数据集和基准测试，本文旨在解决智能模型在理解和推测物体状态变化方面的挑战，提出了一种新的评估多模态大型语言模型 (MLLMs) 的方法，并表明现有的模型在理解这些变化方面仍需要进一步改进。

OSCaR：对象状态字幕和状态变化表示

OSCaR: Object State Captioning and State Change Representation

Explainability of intelligent models has been garnering increasing attention
in recent years. Of the various explainability approaches, concept-based
techniques are notable for utilizing a set of human-meaningful concepts instead
of focusing on individual pixels. However, there is a scarcity of methods that
consistently provide both local and global explanations. Moreover, most of the
methods have no offer to explain misclassification cases. To address these
challenges, our study follows a straightforward yet effective approach. We
propose a unified concept-based system, which inputs a number of
super-pixelated images into the networks, allowing them to learn better
representations of the target's objects as well as the target's concepts. This
method automatically learns, scores, and extracts local and global concepts.
Our experiments revealed that, in addition to enhancing performance, the models
could provide deeper insights into predictions and elucidate false
classifications.

本文提出了一个新的概念性系统，利用一定数量的超像素图像作为输入，可以自动学习、评分和提取局部和全局概念，在提高性能的同时，深入了解预测并阐明误分类。