Multimodal Large Language Models (MLLMs) have made notable advances in visual understanding, yet their abilities to recognize objects modified by specific attributes remain an open question. To address this, we explore MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios. We introduce a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types. We assess 26 recent open-sourced and commercial models using our benchmark. The findings highlight pronounced performance gaps in recognizing objects in NEMO and reveal distinct answer preferences across different models. Although stronger vision encoders improve performance, MLLMs still lag behind standalone vision encoders. Interestingly, scaling up the model size does not consistently yield better outcomes, as deeper analysis reveals that larger LLMs can weaken vision encoders during fine-tuning. These insights shed light on critical limitations in current MLLMs and suggest potential pathways toward developing more versatile and resilient multimodal models.

本文研究了多模态大型语言模型在识别被特定属性修改的物体方面的能力，填补了这一研究空白。我们提出了一个新的基准NEMO，评估了26种近期开源和商业模型的表现，发现这些模型在准确识别属性修改物体上存在显著性能差距，并指出模型规模扩大可能会削弱视觉编码器的表现。这一发现为未来多模态模型的发展提供了重要启示。

NEMO: 多模态大型语言模型能否识别属性修改的物体？