We present GSDeformer, a method that achieves free-form deformation on 3D
Gaussian Splatting(3DGS) without requiring any architectural changes. Our
method extends cage-based deformation, a traditional mesh deformation method,
to 3DGS. This is done by converting 3DGS into a novel proxy point cloud
representation, where its deformation can be used to infer the transformations
to apply on the 3D gaussians making up 3DGS. We also propose an automatic cage
construction algorithm for 3DGS to minimize manual work. Our method does not
modify the underlying architecture of 3DGS. Therefore, any existing trained
vanilla 3DGS can be easily edited by our method. We compare the deformation
capability of our method against other existing methods, demonstrating the ease
of use and comparable quality of our method, despite being more direct and thus
easier to integrate with other concurrent developments on 3DGS.

我们提出了 GSDeformer，这是一种在不需要任何架构变动的情况下，在 3D Gaussian Splatting (3DGS) 上实现自由形变的方法。我们的方法将传统的网格变形方法 (基于 cage 的变形) 扩展到了 3DGS 上，通过将 3DGS 转换为新型的代理点云表示形式，可以推断出应用于构成 3DGS 的 3D 高斯函数的变换。同时，我们还提出了一种自动构建 3DGS cage 的算法，以减少手动工作。我们的方法不修改 3DGS 的基本架构，因此任何现有的已经训练好的标准 3DGS 都可以轻松地通过我们的方法进行编辑。通过与其他现有方法比较我们的方法的变形能力，我们证明了我们的方法的易于使用和可比较的质量，尽管更加直接，因此更容易与 3DGS 上的其他并行开发集成。

GSDeformer：基于直接笼式变形的三维高斯点云投影

GSDeformer: Direct Cage-based Deformation for 3D Gaussian Splatting

Representing 3D shape in deep learning frameworks in an accurate, efficient
and compact manner still remains an open challenge. Most existing work
addresses this issue by employing voxel-based representations. While these
approaches benefit greatly from advances in computer vision by generalizing 2D
convolutions to the 3D setting, they also have several considerable drawbacks.
The computational complexity of voxel-encodings grows cubically with the
resolution thus limiting such representations to low-resolution 3D
reconstruction. In an attempt to solve this problem, point cloud
representations have been proposed. Although point clouds are more efficient
than voxel representations as they only cover surfaces rather than volumes,
they do not encode detailed geometric information about relationships between
points. In this paper we propose a method to learn free-form deformations (FFD)
for the task of 3D reconstruction from a single image. By learning to deform
points sampled from a high-quality mesh, our trained model can be used to
produce arbitrarily dense point clouds or meshes with fine-grained geometry. We
evaluate our proposed framework on both synthetic and real-world data and
achieve state-of-the-art results on point-cloud and volumetric metrics.
Additionally, we qualitatively demonstrate its applicability to label
transferring for 3D semantic segmentation.

本文提出了一种基于自由形变（FFD）的学习方法，用于从单个图像中重建 3D 形状，实现了细节几何信息的精确编码，并在点云和体积度量方面取得了最先进的结果，同时展示了其在 3D 语义分割中的应用。

学习自由形变，用于三维物体重建

Learning Free-Form Deformations for 3D Object Reconstruction

One challenge that remains open in 3D deep learning is how to efficiently
represent 3D data to feed deep networks. Recent works have relied on volumetric
or point cloud representations, but such approaches suffer from a number of
issues such as computational complexity, unordered data, and lack of finer
geometry. This paper demonstrates that a mesh representation (i.e. vertices and
faces to form polygonal surfaces) is able to capture fine-grained geometry for
3D reconstruction tasks. A mesh however is also unstructured data similar to
point clouds. We address this problem by proposing a learning framework to
infer the parameters of a compact mesh representation rather than learning from
the mesh itself. This compact representation encodes a mesh using free-form
deformation and a sparse linear combination of models allowing us to
reconstruct 3D meshes from single images. In contrast to prior work, we do not
rely on silhouettes and landmarks to perform 3D reconstruction. We evaluate our
method on synthetic and real-world datasets with very promising results. Our
framework efficiently reconstructs 3D objects in a low-dimensional way while
preserving its important geometrical aspects.

本文提出了一种使用基于网格的表示来捕捉三维重构任务中的细粒度几何信息的学习框架，它使用自由形变和稀疏线性组合的紧凑网格表示来重建三维物体，相比之前的工作，我们不依赖于轮廓和标记来进行三维重建，并在合成和真实数据集上进行了很有前途的实验结果验证。

Image2Mesh: 单张图像三维重建的学习框架

Image2Mesh: A Learning Framework for Single Image 3D Reconstruction

3D reconstruction from a single image is a key problem in multiple
applications ranging from robotic manipulation to augmented reality. Prior
methods have tackled this problem through generative models which predict 3D
reconstructions as voxels or point clouds. However, these methods can be
computationally expensive and miss fine details. We introduce a new
differentiable layer for 3D data deformation and use it in DeformNet to learn a
model for 3D reconstruction-through-deformation. DeformNet takes an image
input, searches the nearest shape template from a database, and deforms the
template to match the query image. We evaluate our approach on the ShapeNet
dataset and show that - (a) the Free-Form Deformation layer is a powerful new
building block for Deep Learning models that manipulate 3D data (b) DeformNet
uses this FFD layer combined with shape retrieval for smooth and
detail-preserving 3D reconstruction of qualitatively plausible point clouds
with respect to a single query image (c) compared to other state-of-the-art 3D
reconstruction methods, DeformNet quantitatively matches or outperforms their
benchmarks by significant margins. For more information, visit:
this https URL .

本论文提出了一种新的深度学习模型 DeformNet，利用自由形变层和形状检索来实现从单张图片中进行 3D 重建，相较于其他现有方法有更好的表现。