Recent advancements in multimodal large language models (LLMs) have shown
their potential in various domains, especially concept reasoning. Despite these
developments, applications in understanding 3D environments remain limited.
This paper introduces Reason3D, a novel LLM designed for comprehensive 3D
understanding. Reason3D takes point cloud data and text prompts as input to
produce textual responses and segmentation masks, facilitating advanced tasks
like 3D reasoning segmentation, hierarchical searching, express referring, and
question answering with detailed mask outputs. Specifically, we propose a
hierarchical mask decoder to locate small objects within expansive scenes. This
decoder initially generates a coarse location estimate covering the object's
general area. This foundational estimation facilitates a detailed,
coarse-to-fine segmentation strategy that significantly enhances the precision
of object identification and segmentation. Experiments validate that Reason3D
achieves remarkable results on large-scale ScanNet and Matterport3D datasets
for 3D express referring, 3D question answering, and 3D reasoning segmentation
tasks. Code and models are available at:
this https URL

Reason3D 是一种新型的多模态大型语言模型，通过点云数据和文本提示作为输入，生成文本回答和分割遮罩，实现 3D 推理分割、分层搜索、精确引用和问题回答等高级任务。

Reason3D：基于大规模语言模型的 3D 分割搜索和推理

Reason3D: Searching and Reasoning 3D Segmentation via Large Language  Model

The Segment Anything Model (SAM), a profound vision foundation model
pre-trained on a large-scale dataset, breaks the boundaries of general
segmentation and sparks various downstream applications. This paper introduces
Hi-SAM, a unified model leveraging SAM for hierarchical text segmentation.
Hi-SAM excels in text segmentation across four hierarchies, including stroke,
word, text-line, and paragraph, while realizing layout analysis as well.
Specifically, we first turn SAM into a high-quality text stroke segmentation
(TSS) model through a parameter-efficient fine-tuning approach. We use this TSS
model to iteratively generate the text stroke labels in a semi-automatical
manner, unifying labels across the four text hierarchies in the HierText
dataset. Subsequently, with these complete labels, we launch the end-to-end
trainable Hi-SAM based on the TSS architecture with a customized hierarchical
mask decoder. During inference, Hi-SAM offers both automatic mask generation
(AMG) mode and promptable segmentation mode. In terms of the AMG mode, Hi-SAM
segments text stroke foreground masks initially, then samples foreground points
for hierarchical text mask generation and achieves layout analysis in passing.
As for the promptable mode, Hi-SAM provides word, text-line, and paragraph
masks with a single point click. Experimental results show the state-of-the-art
performance of our TSS model: 84.86% fgIOU on Total-Text and 88.96% fgIOU on
TextSeg for text stroke segmentation. Moreover, compared to the previous
specialist for joint hierarchical detection and layout analysis on HierText,
Hi-SAM achieves significant improvements: 4.73% PQ and 5.39% F1 on the
text-line level, 5.49% PQ and 7.39% F1 on the paragraph level layout analysis,
requiring 20x fewer training epochs. The code is available at
this https URL

使用 Segment Anything Model（SAM）为基础，本论文介绍了 Hi-SAM，一个利用 SAM 进行分层文本分割的统一模型，能够在四个层次（笔画、词、文本行和段落）上实现文本分割并进行布局分析。