Manipulation relationship detection (MRD) aims to guide the robot to grasp
objects in the right order, which is important to ensure the safety and
reliability of grasping in object stacked scenes. Previous works infer
manipulation relationship by deep neural network trained with data collected
from a predefined view, which has limitation in visual dislocation in
unstructured environments. Multi-view data provide more comprehensive
information in space, while a challenge of multi-view MRD is domain shift. In
this paper, we propose a novel multi-view fusion framework, namely multi-view
MRD network (MMRDN), which is trained by 2D and 3D multi-view data. We project
the 2D data from different views into a common hidden space and fit the
embeddings with a set of Von-Mises-Fisher distributions to learn the consistent
representations. Besides, taking advantage of position information within the
3D data, we select a set of $K$ Maximum Vertical Neighbors (KMVN) points from
the point cloud of each object pair, which encodes the relative position of
these two objects. Finally, the features of multi-view 2D and 3D data are
concatenated to predict the pairwise relationship of objects. Experimental
results on the challenging REGRAD dataset show that MMRDN outperforms the
state-of-the-art methods in multi-view MRD tasks. The results also demonstrate
that our model trained by synthetic data is capable to transfer to real-world
scenarios.

本文提出了一种基于多视图数据的新型多视图 MRD 网络（MMRDN），该网络通过 2D 和 3D 多视图数据进行训练，结合了 KMVN 和 Von-Mises-Fisher 分布来预测对象间的关系，实验结果表明，该模型能够在多视图 MRD 任务中胜过现有的 state-of-the-art 方法，并且能够将模型从合成数据中训练转移到真实世界情景中。