In the realm of future home-assistant robots, 3D articulated object
manipulation is essential for enabling robots to interact with their
environment. Many existing studies make use of 3D point clouds as the primary
input for manipulation policies. However, this approach encounters challenges
due to data sparsity and the significant cost associated with acquiring point
cloud data, which can limit its practicality. In contrast, RGB images offer
high-resolution observations using cost effective devices but lack spatial 3D
geometric information. To overcome these limitations, we present a novel
image-based robotic manipulation framework. This framework is designed to
capture multiple perspectives of the target object and infer depth information
to complement its geometry. Initially, the system employs an eye-on-hand RGB
camera to capture an overall view of the target object. It predicts the initial
depth map and a coarse affordance map. The affordance map indicates actionable
areas on the object and serves as a constraint for selecting subsequent
viewpoints. Based on the global visual prior, we adaptively identify the
optimal next viewpoint for a detailed observation of the potential manipulation
success area. We leverage geometric consistency to fuse the views, resulting in
a refined depth map and a more precise affordance map for robot manipulation
decisions. By comparing with prior works that adopt point clouds or RGB images
as inputs, we demonstrate the effectiveness and practicality of our method. In
the project webpage (this https URL), real world
experiments further highlight the potential of our method for practical
deployment.

基于图像的机器人操纵系统，利用多个视角捕捉目标物体，推断深度信息以补充其几何信息，采用几何一致性融合视角，实现精确操纵决策。