Recent advances have demonstrated that Language Vision Models (LVMs) surpass
the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision
tasks, motivating attempts to apply LVMs to three-dimensional (3D) data. While
LVMs are efficient and effective in addressing various downstream 2D vision
tasks without training, they face significant challenges when it comes to point
clouds, a representative format for representing 3D data. It is more difficult
to extract features from 3D data and there are challenges due to large data
sizes and the cost of the collection and labelling, resulting in a notably
limited availability of datasets. Moreover, constructing LVMs for point clouds
is even more challenging due to the requirements for large amounts of data and
training time. To address these issues, our research aims to 1) apply the
Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2)
experiment with synthetic data to evaluate its effectiveness in bridging the
gap between synthetic and real-world data domains. Our approach exhibited high
performance with an accuracy of 0.96, an IoU of 0.85, precision of 0.92, recall
of 0.91, and an F1 score of 0.92, confirming its potential. However, challenges
such as occlusion problems and pixel-level overlaps of multi-label points
during spherical image generation remain to be addressed in future studies.

最近的研究进展表明，语言视觉模型在二维计算机视觉任务方面已经超过了现有的最先进技术 (SOTA)，激发了将语言视觉模型应用于三维数据的尝试。然而，应用于点云的 LVM 面临着从三维数据提取特征的困难以及由于数据量大、采集和标注成本高导致数据集的可用性明显有限等挑战。为了解决这些问题，我们的研究旨在通过球面投影将 3D 数据转化为 2D，并尝试使用合成数据评估其在合成和真实数据之间弥合差距方面的效果。我们的方法表现出了高性能，准确率为 0.96，IoU 为 0.85，精确率为 0.92，召回率为 0.91，F1 分数为 0.92，验证了其潜力。然而，在未来的研究中仍需要解决遮挡问题和球面图像生成过程中多标签点的像素级重叠等挑战。

使用语言视觉模型在移动 LiDAR 中进行建筑零击探测

Zero-shot detection of buildings in mobile LiDAR using Language Vision  Model

We introduce Grounded SAM, which uses Grounding DINO as an open-set object
detector to combine with the segment anything model (SAM). This integration
enables the detection and segmentation of any regions based on arbitrary text
inputs and opens a door to connecting various vision models. As shown in Fig.1,
a wide range of vision tasks can be achieved by using the versatile Grounded
SAM pipeline. For example, an automatic annotation pipeline based solely on
input images can be realized by incorporating models such as BLIP and Recognize
Anything. Additionally, incorporating Stable-Diffusion allows for controllable
image editing, while the integration of OSX facilitates promptable 3D human
motion analysis. Grounded SAM also shows superior performance on
open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in
the wild) zero-shot benchmark with the combination of Grounding DINO-Base and
SAM-Huge models.

引入地面 SAM，使用地面化 DINO 作为开放目标检测器并结合分段任何模型（SAM），实现基于任意文本输入的任何区域的检测和分割，并打开了连接各种视觉模型的大门。