We introduce TransformerFusion, a transformer-based 3D scene reconstruction
approach. From an input monocular RGB video, the video frames are processed by
a transformer network that fuses the observations into a volumetric feature
grid representing the scene; this feature grid is then decoded into an implicit
3D scene representation. Key to our approach is the transformer architecture
that enables the network to learn to attend to the most relevant image frames
for each 3D location in the scene, supervised only by the scene reconstruction
task. Features are fused in a coarse-to-fine fashion, storing fine-level
features only where needed, requiring lower memory storage and enabling fusion
at interactive rates. The feature grid is then decoded to a higher-resolution
scene reconstruction, using an MLP-based surface occupancy prediction from
interpolated coarse-to-fine 3D features. Our approach results in an accurate
surface reconstruction, outperforming state-of-the-art multi-view stereo depth
estimation methods, fully-convolutional 3D reconstruction approaches, and
approaches using LSTM- or GRU-based recurrent networks for video sequence
fusion.

TransformerFusion 是一种基于 Transformer 的 3D 场景重建方法，通过 Transformer 网络处理输入的单眼 RGB 视频，将观察结果融合为代表场景的体积特征网格，最终解码为隐式的 3D 场景表示，实现精准的场景面貌恢复与重建。