Multi-view projection techniques have shown themselves to be highly effective
in achieving top-performing results in the recognition of 3D shapes. These
methods involve learning how to combine information from multiple view-points.
However, the camera view-points from which these views are obtained are often
fixed for all shapes. To overcome the static nature of current multi-view
techniques, we propose learning these view-points. Specifically, we introduce
the Multi-View Transformation Network (MVTN), which uses differentiable
rendering to determine optimal view-points for 3D shape recognition. As a
result, MVTN can be trained end-to-end with any multi-view network for 3D shape
classification. We integrate MVTN into a novel adaptive multi-view pipeline
that is capable of rendering both 3D meshes and point clouds. Our approach
demonstrates state-of-the-art performance in 3D classification and shape
retrieval on several benchmarks (ModelNet40, ScanObjectNN, ShapeNet Core55).
Further analysis indicates that our approach exhibits improved robustness to
occlusion compared to other methods. We also investigate additional aspects of
MVTN, such as 2D pretraining and its use for segmentation. To support further
research in this area, we have released MVTorch, a PyTorch library for 3D
understanding and generation using multi-view projections.

本研究提出了一种名为 MVTN 的多视图转换网络，利用可微分渲染确定最佳视点进行三维形状识别，将其集成到新颖的自适应多视图流程中，在几个基准测验中表现出 3D 分类和形状检索的最先进性能，并展示了相对于其他方法更好的遮挡鲁棒性，并探讨了 MVTN 的其他方面，如二维预训练和其用于分割，同时公开了使用多视图投影进行 3D 理解和生成的 PyTorch 库 MVTorch。

MVTN：学习多视角转换的三维理解

MVTN: Learning Multi-View Transformations for 3D Understanding

Multi-view projection methods have demonstrated their ability to reach
state-of-the-art performance on 3D shape recognition. Those methods learn
different ways to aggregate information from multiple views. However, the
camera view-points for those views tend to be heuristically set and fixed for
all shapes. To circumvent the lack of dynamism of current multi-view methods,
we propose to learn those view-points. In particular, we introduce the
Multi-View Transformation Network (MVTN) that regresses optimal view-points for
3D shape recognition, building upon advances in differentiable rendering. As a
result, MVTN can be trained end-to-end along with any multi-view network for 3D
shape classification. We integrate MVTN in a novel adaptive multi-view pipeline
that can render either 3D meshes or point clouds. MVTN exhibits clear
performance gains in the tasks of 3D shape classification and 3D shape
retrieval without the need for extra training supervision. In these tasks, MVTN
achieves state-of-the-art performance on ModelNet40, ShapeNet Core55, and the
most recent and realistic ScanObjectNN dataset (up to 6% improvement).
Interestingly, we also show that MVTN can provide network robustness against
rotation and occlusion in the 3D domain. The code is available at
this https URL .

本文介绍了一种基于多角度变换的深度学习网络架构，通过改变视角，以使得机器在 3D 形状识别任务中取得更好的性能。该网络框架在静态视角方法的基础上，实现了自适应动态视角模式，将深度学习与可微渲染技术相结合，并在三个公开数据集上取得了较高的识别精度。