We study the problem of translating an image-based, step-by-step assembly
manual created by human designers into machine-interpretable instructions. We
formulate this problem as a sequential prediction task: at each step, our model
reads the manual, locates the components to be added to the current shape, and
infers their 3D poses. This task poses the challenge of establishing a 2D-3D
correspondence between the manual image and the real 3D object, and 3D pose
estimation for unseen 3D objects, since a new component to be added in a step
can be an object built from previous steps. To address these two challenges, we
present a novel learning-based framework, the Manual-to-Executable-Plan Network
(MEPNet), which reconstructs the assembly steps from a sequence of manual
images. The key idea is to integrate neural 2D keypoint detection modules and
2D-3D projection algorithms for high-precision prediction and strong
generalization to unseen components. The MEPNet outperforms existing methods on
three newly collected LEGO manual datasets and a Minecraft house dataset.

本文研究的是将由人设计师创建的基于图像的逐步组装手册翻译成为机器可解释指令的问题，采用一种基于学习的框架，即 MEPNet，该框架通过对逐步手册图像序列的重建来重建组合步骤，通过神经网络的 2D 关键点检测模块和 2D-3D 投影算法来预测 3D 姿态。