Generating 3D models from multi-view 2D RGB images has gained significant attention, extending the capabilities of technologies like Virtual Reality, Robotic Vision, and human-machine interaction. In this paper, we introduce a hybrid strategy combining CNNs and transformers, featuring a visual auto-encoder with self-attention mechanisms and a 3D refiner network, trained using a novel Joint Train Separate Optimization (JTSO) algorithm. Encoded features from unordered inputs are transformed into an enhanced feature map by the self-attention layer, decoded into an initial 3D volume, and further refined. Our network generates 3D voxels from single or multiple 2D images from arbitrary viewpoints. Performance evaluations using the ShapeNet datasets show that our approach, combined with JTSO, outperforms state-of-the-art techniques in single and multi-view 3D reconstruction, achieving the highest mean intersection over union (IOU) scores, surpassing other models by 4.2% in single-view reconstruction.

本研究解决了从多视角2D RGB图像生成3D模型的技术瓶颈，通过结合CNN和变换器的混合策略，提出了使用自注意力机制的视觉自编码器和3D精炼网络。研究表明，使用新颖的联合训练分开优化(JTSO)算法，所提出的方法在单视图和多视图3D重建中达到了最高的平均交集比(IOU)得分，超越了其他模型4.2%。

Refine3DNet：通过关注机制提高多视角RGB图像的3D物体重建精度