We present Real2Code, a novel approach to reconstructing articulated objects
via code generation. Given visual observations of an object, we first
reconstruct its part geometry using an image segmentation model and a shape
completion model. We then represent the object parts with oriented bounding
boxes, which are input to a fine-tuned large language model (LLM) to predict
joint articulation as code. By leveraging pre-trained vision and language
models, our approach scales elegantly with the number of articulated parts, and
generalizes from synthetic training data to real world objects in unstructured
environments. Experimental results demonstrate that Real2Code significantly
outperforms previous state-of-the-art in reconstruction accuracy, and is the
first approach to extrapolate beyond objects' structural complexity in the
training set, and reconstructs objects with up to 10 articulated parts. When
incorporated with a stereo reconstruction model, Real2Code also generalizes to
real world objects from a handful of multi-view RGB images, without the need
for depth or camera information.

通过代码生成，使用 Real2Code 方法对关节物体进行重建，超过先前最先进的重建准确度，并且在训练集的结构复杂性之外，首次对具有多达 10 个关节部件的物体进行重建。