PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address
the segmentation task challenges. To overcome the limitation of the LMM being
limited to textual output, PSALM incorporates a mask decoder and a
well-designed input schema to handle a variety of segmentation tasks. This
schema includes images, task instructions, conditional prompts, and mask
tokens, which enable the model to generate and classify segmentation masks
effectively. The flexible design of PSALM supports joint training across
multiple datasets and tasks, leading to improved performance and task
generalization. PSALM achieves superior results on several benchmarks, such as
RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive,
and further exhibits zero-shot capabilities on unseen tasks, such as
open-vocabulary segmentation, generalized referring expression segmentation and
video object segmentation, making a significant step towards a GPT moment in
computer vision. Through extensive experiments, PSALM demonstrates its
potential to transform the domain of image segmentation, leveraging the robust
visual understanding capabilities of LMMs as seen in natural language
processing. Code and models are available at this https URL

PSALM 是基于大型多模态模型（LMM）的强大扩展，用于解决分割任务的挑战，通过引入掩码解码器和精心设计的输入模式来处理各种分割任务，并且能够生成和分类分割掩码，支持多个数据集和任务的联合训练，具有卓越的性能和任务泛化能力，对图像分割领域具有潜力。

PSALM: 基于大型多模型的逐像素分割

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

Large Multimodal Model (LMM) is a hot research topic in the computer vision
area and has also demonstrated remarkable potential across multiple
disciplinary fields. A recent trend is to further extend and enhance the
perception capabilities of LMMs. The current methods follow the paradigm of
adapting the visual task outputs to the format of the language model, which is
the main component of a LMM. This adaptation leads to convenient development of
such LMMs with minimal modifications, however, it overlooks the intrinsic
characteristics of diverse visual tasks and hinders the learning of perception
capabilities. To address this issue, we propose a novel LMM architecture named
Lumen, a Large multimodal model with versatile vision-centric capability
enhancement. We decouple the LMM's learning of perception capabilities into
task-agnostic and task-specific stages. Lumen first promotes fine-grained
vision-language concept alignment, which is the fundamental capability for
various visual tasks. Thus the output of the task-agnostic stage is a shared
representation for all the tasks we address in this paper. Then the
task-specific decoding is carried out by flexibly routing the shared
representation to lightweight task decoders with negligible training efforts.
Benefiting from such a decoupled design, our Lumen surpasses existing LMM-based
approaches on the COCO detection benchmark with a clear margin and exhibits
seamless scalability to additional visual tasks. Furthermore, we also conduct
comprehensive ablation studies and generalization evaluations for deeper
insights. The code will be released at this https URL

大型多模态模型（LMM）是计算机视觉领域的热门研究课题，近期的趋势是进一步拓展和增强 LMM 的感知能力。我们提出了一种名为 Lumen 的新型 LMM 架构，将 LMM 的感知能力学习分解为任务无关和任务特定阶段，在 COCO 检测基准上显著超越现有的基于 LMM 的方法，并展现了对其他视觉任务的无缝可扩展性。

Lumen: 开放大型多模态模型的多样视觉能力

Lumen: Unleashing Versatile Vision-Centric Capabilities of Large  Multimodal Models

This paper presents a comprehensive evaluation of the Optical Character
Recognition (OCR) capabilities of the recently released GPT-4V(ision), a Large
Multimodal Model (LMM). We assess the model's performance across a range of OCR
tasks, including scene text recognition, handwritten text recognition,
handwritten mathematical expression recognition, table structure recognition,
and information extraction from visually-rich document. The evaluation reveals
that GPT-4V performs well in recognizing and understanding Latin contents, but
struggles with multilingual scenarios and complex tasks. Based on these
observations, we delve deeper into the necessity of specialized OCR models and
deliberate on the strategies to fully harness the pretrained general LMMs like
GPT-4V for OCR downstream tasks. The study offers a critical reference for
future research in OCR with LMMs. Evaluation pipeline and results are available
at this https URL

GPT-4V 综合评估了最近发布的大型多模态模型 (GPT-4V (ision)) 在光学字符识别 (OCR) 方面的能力，发现其在识别和理解拉丁内容方面表现良好，但在多语言情境和复杂任务中存在困难。基于这些观察，我们深入探讨了专门的 OCR 模型的必要性，以及充分利用预训练的通用 LMM 模型如 GPT-4V 来进行 OCR 下游任务的策略。该研究为未来 LMM 在 OCR 领域的研究提供了重要参考。