Generating 3D shapes from single RGB images is essential in various applications such as robotics. Current approaches typically target images containing clear and complete visual descriptions of the object, without considering common realistic cases where observations of objects that are largely occluded or truncated. We thus propose a transformer-based autoregressive model to generate the probabilistic distribution of 3D shapes conditioned on an RGB image containing potentially highly ambiguous observations of the object. To handle realistic scenarios such as occlusion or field-of-view truncation, we create simulated image-to-shape training pairs that enable improved fine-tuning for real-world scenarios. We then adopt cross-attention to effectively identify the most relevant region of interest from the input image for shape generation. This enables inference of sampled shapes with reasonable diversity and strong alignment with the input image. We train and test our model on our synthetic data then fine-tune and test it on real-world data. Experiments demonstrate that our model outperforms state of the art in both scenarios

提出了一种基于Transformer的自回归模型，根据可能以高度模糊的观测图像为基础的RGB图像，生成3D形状的概率分布，该模型采用交叉注意力机制，有效地识别形状生成的最相关兴趣区域，并在合成数据和真实数据上得到优于现有方法的结果。

PT43D：基于单一高度模糊 RGB 图像生成三维形状的概率变换器