Online construction of open-ended language scenes is crucial for robotic
applications, where open-vocabulary interactive scene understanding is
required. Recently, neural implicit representation has provided a promising
direction for online interactive mapping. However, implementing open-vocabulary
scene understanding capability into online neural implicit mapping still faces
three challenges: lack of local scene updating ability, blurry spatial
hierarchical semantic segmentation and difficulty in maintaining multi-view
consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based
language and geometric features to create an open-vocabulary field, thus
allowing for local updates during online training process. Additionally, we
leverage a foundational model for image segmentation to extract language
features on object-level entities, achieving clear segmentation boundaries and
hierarchical semantic features. For the purpose of preserving consistency in 3D
object properties across different viewpoints, we propose a spatial adaptive
voxel adjustment mechanism and a multi-view weight selection method. Extensive
experiments on open-vocabulary object localization and semantic segmentation
demonstrate that O2V-mapping achieves online construction of language scenes
while enhancing accuracy, outperforming the previous SOTA method.

O2V-mapping utilizes voxel-based language and geometric features to achieve online construction of open-vocabulary language scenes, overcoming challenges such as lack of local scene updating ability, blurry spatial hierarchical semantic segmentation, and difficulty in maintaining multi-view consistency.

O2V-Mapping：基于神经隐性表示的在线开放词汇映射

O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit  Representation

Humans are able to recognize structured relations in observation, allowing us
to decompose complex scenes into simpler parts and abstract the visual world in
multiple levels. However, such hierarchical reasoning ability of human
perception remains largely unexplored in current literature of semantic
segmentation. Existing work is often aware of flatten labels and predicts
target classes exclusively for each pixel. In this paper, we instead address
hierarchical semantic segmentation (HSS), which aims at structured, pixel-wise
description of visual observation in terms of a class hierarchy. We devise
HSSN, a general HSS framework that tackles two critical issues in this task: i)
how to efficiently adapt existing hierarchy-agnostic segmentation networks to
the HSS setting, and ii) how to leverage the hierarchy information to
regularize HSS network learning. To address i), HSSN directly casts HSS as a
pixel-wise multi-label classification task, only bringing minimal architecture
change to current segmentation models. To solve ii), HSSN first explores
inherent properties of the hierarchy as a training objective, which enforces
segmentation predictions to obey the hierarchy structure. Further, with
hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space,
so as to generate well-structured pixel representations and improve
segmentation eventually. We conduct experiments on four semantic segmentation
datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, and PASCAL-Person-Part),
with different class hierarchies, segmentation network architectures and
backbones, showing the generalization and superiority of HSSN.

本文提出了一种用于分层语义分割的新架构 HSSN，它可将现有的分割网络有效地应用于分层语义分割，通过像素嵌入和类别层次结构来实现像素级多标签分类和更好的分割。

深层次语义分割

Deep Hierarchical Semantic Segmentation

Structure extraction from document images has been a long-standing research
topic due to its high impact on a wide range of practical applications. In this
paper, we share our findings on employing a hierarchical semantic segmentation
network for this task of structure extraction. We propose a prior based deep
hierarchical CNN network architecture that enables document structure
extraction using very high resolution(1800 x 1000) images. We divide the
document image into overlapping horizontal strips such that the network
segments a strip and uses its prediction mask as prior for predicting the
segmentation of the subsequent strip. We perform experiments establishing the
effectiveness of our strip based network architecture through ablation methods
and comparison with low-resolution variations. Further, to demonstrate our
network's capabilities, we train it on only one type of documents (Forms) and
achieve state-of-the-art results over other general document datasets. We
introduce our new human-annotated forms dataset and show that our method
significantly outperforms different segmentation baselines on this dataset in
extracting hierarchical structures. Our method is currently being used in
Adobe's AEM Forms for automated conversion of paper and PDF forms to modern
HTML based forms.

本研究基于层次语义分割网络，提出了一种基于先验模型的深层次 CNN 网络结构，用于高分辨率文档图像的结构提取，通过在文档图像中将其分成互相重叠的水平条带进行分割，本网络得到了优秀的效果，我们还新建了人工标注的文档表单数据集并在此数据集上进行实验，对比不同分割 baseline 方法证明了本网络算法在分层结构提取上的高效性。当前，此算法已用于 Adobe 的 AEM Forms，实现了纸质和 PDF 表单的自动转换为现代 HTML 表单。