The rapid development of large language and vision models (LLVMs) has been
driven by advances in visual instruction tuning. Recently, open-source LLVMs
have curated high-quality visual instruction tuning datasets and utilized
additional vision encoders or multiple computer vision models in order to
narrow the performance gap with powerful closed-source LLVMs. These
advancements are attributed to multifaceted information required for diverse
capabilities, including fundamental image understanding, real-world knowledge
about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
signs, and math problems), and step-by-step procedures for solving complex
questions. Drawing from the multifaceted information, we present a new
efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
multifaceted rationale to enhance understanding and answering capabilities. To
embed lengthy rationales containing abundant information, we employ the Mamba
architecture, capable of processing sequential data with linear time
complexity. We introduce a new concept of traversal of rationale that
facilitates efficient embedding of rationale. Subsequently, the backbone
multimodal language model (MLM) is trained to generate answers with the aid of
rationale. Through these steps, Meteor achieves significant improvements in
vision language performances across multiple evaluation benchmarks requiring
diverse capabilities, without scaling up the model size or employing additional
vision encoders and computer vision models.

通过使用多面手的理由，我们提出了一种新的有效率的 LLVM（Mamba），以提高理解和回答能力，并在多个评估基准中取得了显著的视觉语言性能改进。

Meteor：基于曼巴的对大型语言和视觉模型注释的遍历

Meteor: Mamba-based Traversal of Rationale for Large Language and Vision  Models

In this paper, we discuss approaches for integrating Computational Creativity
(CC) with research in large language and vision models (LLVMs) to address a key
limitation of these models, i.e., creative problem solving. We present
preliminary experiments showing how CC principles can be applied to address
this limitation through augmented prompting. With this work, we hope to foster
discussions of Computational Creativity in the context of ML algorithms for
creative problem solving in LLVMs. Our code is at:
this https URL

本文讨论了将计算创造力与大语言和视觉模型集成以解决创新问题解决的关键局限性，通过增强提示的方式初步实验展示了计算创造力原则在此方面的应用，旨在促进对计算创造力在大语言和视觉模型中的创新问题解决的机器学习算法的讨论。

大型语言和视觉模型中的创新问题解决 -- 需要什么？

Creative Problem Solving in Large Language and Vision Models -- What  Would it Take?

The recent success of large language and vision models on vision question
answering (VQA), particularly their applications in medicine (Med-VQA), has
shown a great potential of realizing effective visual assistants for
healthcare. However, these models are not extensively tested on the
hallucination phenomenon in clinical settings. Here, we created a hallucination
benchmark of medical images paired with question-answer sets and conducted a
comprehensive evaluation of the state-of-the-art models. The study provides an
in-depth analysis of current models limitations and reveals the effectiveness
of various prompting strategies.

最近大型语言和视觉模型在视觉问答（VQA）方面取得的成功，特别是在医学领域的应用（Med-VQA），表明实现有效的医疗视觉助手具有巨大潜力。然而，这些模型在临床环境中并没有经过广泛测试，我们在这里创建了一种医学图像的幻觉基准，并对最先进的模型进行了全面评估。该研究对当前模型的局限性进行了深入分析，并揭示了各种提示策略的有效性。