Objects manipulated by the hand (i.e., manipulanda) are particularly
challenging to reconstruct from in-the-wild RGB images or videos. Not only does
the hand occlude much of the object, but also the object is often only visible
in a small number of image pixels. At the same time, two strong anchors emerge
in this setting: (1) estimated 3D hands help disambiguate the location and
scale of the object, and (2) the set of manipulanda is small relative to all
possible objects. With these insights in mind, we present a scalable paradigm
for handheld object reconstruction that builds on recent breakthroughs in large
language/vision models and 3D object datasets. Our model, MCC-Hand-Object
(MCC-HO), jointly reconstructs hand and object geometry given a single RGB
image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
a 3D object model that matches the object in the image and rigidly align the
model to the network-inferred geometry; we call this alignment
Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
achieves state-of-the-art performance on lab and Internet datasets, and we show
how RAR can be used to automatically obtain 3D labels for in-the-wild images of
hand-object interactions.

对象手动操纵，手持物体重建，MCC-HO，检索增强重建，三维标签。

重建手持物体的三维

Reconstructing Hand-Held Objects in 3D

Semantic segmentation is an important component in the perception systems of
autonomous vehicles. In this work, we adopt recent advances in both image and
point cloud segmentation to achieve a better accuracy in the task of segmenting
LiDAR scans. KPRNet improves the convolutional neural network architecture of
2D projection methods and utilizes KPConv to replace the commonly used
post-processing techniques with a learnable point-wise component which allows
us to obtain more accurate 3D labels. With these improvements our model
outperforms the current best method on the SemanticKITTI benchmark, reaching an
mIoU of 63.1.

本研究采用最新的图像和点云分割技术，在卷积神经网络体系结构和 point-wise 组件中应用 KPConv，从而在对 LiDAR 扫描进行分割的任务中实现更好的准确性。已经在 SemanticKITTI 基准测试中超过了当前最佳方法，达到了 63.1 的 mIoU。