Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new
avenues for human interaction with embodied agents or robots using natural
language. Despite this progress, we find a notable limitation: existing 3D-VL
models exhibit sensitivity to the styles of language input, struggling to
understand sentences with the same semantic meaning but written in different
variants. This observation raises a critical question: Can 3D vision-language
models truly understand natural language? To test the language
understandability of 3D-VL models, we first propose a language robustness task
for systematically assessing 3D-VL models across various tasks, benchmarking
their performance when presented with different language style variants.
Importantly, these variants are commonly encountered in applications requiring
direct interaction with humans, such as embodied robotics, given the diversity
and unpredictability of human language. We propose a 3D Language Robustness
Dataset, designed based on the characteristics of human language, to facilitate
the systematic study of robustness. Our comprehensive evaluation uncovers a
significant drop in the performance of all existing models across various 3D-VL
tasks. Even the state-of-the-art 3D-LLM fails to understand some variants of
the same sentences. Further in-depth analysis suggests that the existing models
have a fragile and biased fusion module, which stems from the low diversity of
the existing dataset. Finally, we propose a training-free module driven by LLM,
which improves language robustness. Datasets and code will be available at
github.

3D-VL 模型面临语言输入样式的敏感性，本研究通过提出一个语言鲁棒性任务和设计 3D 语言鲁棒性数据集评估现有模型的性能，在各种 3D-VL 任务中发现所有模型性能显著下降。现有模型存在脆弱和偏置的融合模块，缺乏多样性的现有数据集是其根源，最后通过由 LLM 驱动的无需训练模块来提高语言鲁棒性。

三维视觉语言模型真正理解自然语言吗？

Can 3D Vision-Language Models Truly Understand Natural Language?

Precisely perceiving the geometric and semantic properties of real-world 3D
objects is crucial for the continued evolution of augmented reality and robotic
applications. To this end, we present \algfull{} (\algname{}), which
incorporates vision-language embeddings of foundation models into 3D Gaussian
Splatting (GS). The key contribution of this work is an efficient method to
reconstruct and represent 3D vision-language models. This is achieved by
distilling feature maps generated from image-based foundation models into those
rendered from our 3D model. To ensure high-quality rendering and fast training,
we introduce a novel scene representation by integrating strengths from both GS
and multi-resolution hash encodings (MHE). Our effective training procedure
also introduces a pixel alignment loss that makes the rendered feature distance
of same semantic entities close, following the pixel-level semantic boundaries.
Our results demonstrate remarkable multi-view semantic consistency,
facilitating diverse downstream tasks, beating state-of-the-art methods by
$\mathbf{10.2}$ percent on open-vocabulary language-based object detection,
despite that we are $\mathbf{851\times}$ faster for inference. This research
explores the intersection of vision, language, and 3D scene representation,
paving the way for enhanced scene understanding in uncontrolled real-world
environments. We plan to release the code upon paper acceptance.

通过将视觉语言嵌入模型与三维高斯喷洒算法相结合，本研究提出了一种高效的三维视觉语言模型重建方法，通过渲染从三维模型生成的特征映射，实现了高质量的渲染和快速训练，并引入了像素对齐损失来保证渲染的语义一致性，取得了显著的多视图语义一致性，为增强对无控制的现实环境中场景的理解铺平了道路。

FMGS: 嵌入式基础模型的整体三维高斯重叠投射用于全景三维场景理解

FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D  Scene Understanding

Recent advances in Large Multimodal Models (LMM) have made it possible for
various applications in human-machine interactions. However, developing LMMs
that can comprehend, reason, and plan in complex and diverse 3D environments
remains a challenging topic, especially considering the demand for
understanding permutation-invariant point cloud 3D representations of the 3D
scene. Existing works seek help from multi-view images, and project 2D features
to 3D space as 3D scene representations. This, however, leads to huge
computational overhead and performance degradation. In this paper, we present
LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and
respond to both textual-instructions and visual-prompts. This help LMMs better
comprehend human interactions and further help to remove the ambiguities in
cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results,
and surpasses various 3D vision-language models on both 3D Dense Captioning and
3D Question Answering.

LL3DA 是一种大型语言 3D 助手，可以直接接收点云输入，并对文本指令和视觉提示进行回应，可以帮助大型多模态模型更好地理解人类互动，并在杂乱的 3D 场景中消除歧义。实验证明，LL3DA 在 3D 密集字幕和 3D 问答方面取得了显著成果，并超越了各种 3D 视觉语言模型。