While Multimodal Large Language Models (MLLMs) are widely used for a variety
of vision-language tasks, one observation is that they sometimes misinterpret
visual inputs or fail to follow textual instructions even in straightforward
cases, leading to irrelevant responses, mistakes, and ungrounded claims. This
observation is analogous to a phenomenon in neuropsychology known as Agnosia,
an inability to correctly process sensory modalities and recognize things
(e.g., objects, colors, relations). In our study, we adapt this similar concept
to define "agnosia in MLLMs", and our goal is to comprehensively evaluate and
mitigate such agnosia in MLLMs. Inspired by the diagnosis and treatment process
in neuropsychology, we propose a novel framework EMMA (Evaluation and
Mitigation of Multimodal Agnosia). In EMMA, we develop an evaluation module
that automatically creates fine-grained and diverse visual question answering
examples to assess the extent of agnosia in MLLMs comprehensively. We also
develop a mitigation module to reduce agnosia in MLLMs through multimodal
instruction tuning on fine-grained conversations. To verify the effectiveness
of our framework, we evaluate and analyze agnosia in seven state-of-the-art
MLLMs using 9K test samples. The results reveal that most of them exhibit
agnosia across various aspects and degrees. We further develop a fine-grained
instruction set and tune MLLMs to mitigate agnosia, which led to notable
improvement in accuracy.

我们提出了一个名为 EMMA 的框架，用于全面评估和减轻多模态语言模型中的 agnosia，并通过多模态指导调整来减少 agnosia，并证实了该框架的有效性。

多模式大型语言模型中对失知症的评估与缓解

Evaluation and Mitigation of Agnosia in Multimodal Large Language Models

Neural machine translation models usually adopt the teacher forcing strategy
for training which requires the predicted sequence matches ground truth word by
word and forces the probability of each prediction to approach a 0-1
distribution. However, the strategy casts all the portion of the distribution
to the ground truth word and ignores other words in the target vocabulary even
when the ground truth word cannot dominate the distribution. To address the
problem of teacher forcing, we propose a method to introduce an evaluation
module to guide the distribution of the prediction. The evaluation module
accesses each prediction from the perspectives of fluency and faithfulness to
encourage the model to generate the word which has a fluent connection with its
past and future translation and meanwhile tends to form a translation
equivalent in meaning to the source. The experiments on multiple translation
tasks show that our method can achieve significant improvements over strong
baselines.

论文提出了一种引入评估模块来指导预测结果分布的神经机器翻译方法，从流畅性和忠实度的角度引导模型生成具有连接性的短语，实验结果表明，该方法可以显著提高翻译质量。