The capability of intelligent models to extrapolate and comprehend changes in
object states is a crucial yet demanding aspect of AI research, particularly
through the lens of human interaction in real-world settings. This task
involves describing complex visual environments, identifying active objects,
and interpreting their changes as conveyed through language. Traditional
methods, which isolate object captioning and state change detection, offer a
limited view of dynamic environments. Moreover, relying on a small set of
symbolic words to represent changes has restricted the expressiveness of
language. To address these challenges, in this paper, we introduce the Object
State Captioning and State Change Representation (OSCaR) dataset and benchmark.
OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique
objects from various egocentric video collections. It sets a new testbed for
evaluating multimodal large language models (MLLMs). Our experiments
demonstrate that while MLLMs show some skill, they lack a full understanding of
object state changes. The benchmark includes a fine-tuned model that, despite
initial capabilities, requires significant improvements in accuracy and
generalization ability for effective understanding of these changes. Our code
and dataset are available at this https URL

通过介绍 Object State Captioning and State Change Representation (OSCaR) 数据集和基准测试，本文旨在解决智能模型在理解和推测物体状态变化方面的挑战，提出了一种新的评估多模态大型语言模型 (MLLMs) 的方法，并表明现有的模型在理解这些变化方面仍需要进一步改进。