Recently, self-supervised pre-training has advanced Vision Transformers on
various tasks w.r.t. different data modalities, e.g., image and 3D point cloud
data. In this paper, we explore this learning paradigm for 3D mesh data
analysis based on Transformers. Since applying Transformer architectures to new
modalities is usually non-trivial, we first adapt Vision Transformer to 3D mesh
data processing, i.e., Mesh Transformer. In specific, we divide a mesh into
several non-overlapping local patches with each containing the same number of
faces and use the 3D position of each patch's center point to form positional
embeddings. Inspired by MAE, we explore how pre-training on 3D mesh data with
the Transformer-based structure benefits downstream 3D mesh analysis tasks. We
first randomly mask some patches of the mesh and feed the corrupted mesh into
Mesh Transformers. Then, through reconstructing the information of masked
patches, the network is capable of learning discriminative representations for
mesh data. Therefore, we name our method MeshMAE, which can yield
state-of-the-art or comparable performance on mesh analysis tasks, i.e.,
classification and segmentation. In addition, we also conduct comprehensive
ablation studies to show the effectiveness of key designs in our method.

本文探讨了将自监督预训练用于基于 transformers 的 3D 网格数据分析的学习范式，并首先将 Vision Transformer 适应到 3D 网格数据处理中，即 Mesh Transformer，并通过 MAE 的启发，在 3D 网格数据上进行自学习，提出了 MeshMAE 方法，可以提供 3D 网格分析任务的最新性能，并证明了我们方法中关键设计的有效性。

MeshMAE: 用于三维网格数据分析的遮罩自编码器

MeshMAE: Masked Autoencoders for 3D Mesh Data Analysis

We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D
human pose and mesh vertices from a single image. Our method uses a transformer
encoder to jointly model vertex-vertex and vertex-joint interactions, and
outputs 3D joint coordinates and mesh vertices simultaneously. Compared to
existing techniques that regress pose and shape parameters, METRO does not rely
on any parametric mesh models like SMPL, thus it can be easily extended to
other objects such as hands. We further relax the mesh topology and allow the
transformer self-attention mechanism to freely attend between any two vertices,
making it possible to learn non-local relationships among mesh vertices and
joints. With the proposed masked vertex modeling, our method is more robust and
effective in handling challenging situations like partial occlusions. METRO
generates new state-of-the-art results for human mesh reconstruction on the
public Human3.6M and 3DPW datasets. Moreover, we demonstrate the
generalizability of METRO to 3D hand reconstruction in the wild, outperforming
existing state-of-the-art methods on FreiHAND dataset. Code and pre-trained
models are available at this https URL

本研究提出了一种名为 MEsh TRansfOrmer（METRO）的新方法，该方法使用变压器编码器来同时建模顶点 - 顶点和顶点 - 关节交互，并从单个图像中输出 3D 关节坐标和网格顶点，其非依赖于任何参数化网格模型，可以轻松地扩展到其他对象，如手的 3D 重建。该方法在处理局部遮挡等具有挑战性情况方面更加鲁棒和有效，在公共数据集 Human3.6M 和 3DPW 上为人类网格重建生成了最新的最先进结果。此外，我们展示了 METRO 到野外 3D 手重建的泛化能力，在 FreiHAND 数据集上性能优于现有最先进方法。