This paper focuses on the sim-to-real issue of RGB-D grasp detection and
formulates it as a domain adaptation problem. In this case, we present a
global-to-local method to address hybrid domain gaps in RGB and depth data and
insufficient multi-modal feature alignment. First, a self-supervised rotation
pre-training strategy is adopted to deliver robust initialization for RGB and
depth networks. We then propose a global-to-local alignment pipeline with
individual global domain classifiers for scene features of RGB and depth images
as well as a local one specifically working for grasp features in the two
modalities. In particular, we propose a grasp prototype adaptation module,
which aims to facilitate fine-grained local feature alignment by dynamically
updating and matching the grasp prototypes from the simulation and real-world
scenarios throughout the training process. Due to such designs, the proposed
method substantially reduces the domain shift and thus leads to consistent
performance improvements. Extensive experiments are conducted on the
GraspNet-Planar benchmark and physical environment, and superior results are
achieved which demonstrate the effectiveness of our method.

该论文通过领域适应问题将 RGB-D 抓握检测的问题转变为 sim-to-real 问题，并提出了一种全局到局部的方法来解决 RGB 和深度数据之间的差距以及多模态特征对齐的不足。在大量实验中，该方法显著减小了领域转移，并获得了令人满意的结果，证明了其有效性。

基于全局到局部 RGB-D 适应的实境抓取检测

Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation

This paper aims to caption daily life --i.e., to create a textual description
of people's activities and interactions with objects in their homes. Addressing
this problem requires novel methods beyond traditional video captioning, as
most people would have privacy concerns about deploying cameras throughout
their homes. We introduce RF-Diary, a new model for captioning daily life by
analyzing the privacy-preserving radio signal in the home with the home's
floormap. RF-Diary can further observe and caption people's life through walls
and occlusions and in dark settings. In designing RF-Diary, we exploit the
ability of radio signals to capture people's 3D dynamics, and use the floormap
to help the model learn people's interactions with objects. We also use a
multi-modal feature alignment training scheme that leverages existing
video-based captioning datasets to improve the performance of our radio-based
captioning model. Extensive experimental results demonstrate that RF-Diary
generates accurate captions under visible conditions. It also sustains its good
performance in dark or occluded settings, where video-based captioning
approaches fail to generate meaningful captions. For more information, please
visit our project webpage: this http URL

该论文介绍了一种通过分析家庭中的隐私保护无线电信号和楼层图来实现对日常生活进行文字描述的新模型 RF-Diary，可以穿过墙壁或障碍物，在暗条件下持续准确描绘人们的行动和物品交互，相对于传统视频字幕，具有更好的隐私保护。