Touch and vision go hand in hand, mutually enhancing our ability to
understand the world. From a research perspective, the problem of mixing touch
and vision is underexplored and presents interesting challenges. To this end,
we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data
(local depth maps) with multi-view vision data to achieve surface
reconstruction and novel view synthesis. Our method optimises 3D Gaussian
primitives to accurately model the object's geometry at points of contact. By
creating a framework that decreases the transmittance at touch locations, we
achieve a refined surface reconstruction, ensuring a uniformly smooth depth
map. Touch is particularly useful when considering non-Lambertian objects (e.g.
shiny or reflective surfaces) since contemporary methods tend to fail to
reconstruct with fidelity specular highlights. By combining vision and tactile
sensing, we achieve more accurate geometry reconstructions with fewer images
than prior methods. We conduct evaluation on objects with glossy and reflective
surfaces and demonstrate the effectiveness of our approach, offering
significant improvements in reconstruction quality.

通过将触觉数据与多视角视觉数据相结合，使用触觉引导的 3DGS 方法进行表面重建和新视角合成，以实现更准确的几何重建。

Snap-it, Tap-it, Splat-it: 用于重建具有挑战性表面的触觉感知的三维高斯散点

Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for  Reconstructing Challenging Surfaces

Implicit neural networks have emerged as a crucial technology in 3D surface
reconstruction. To reconstruct continuous surfaces from discrete point clouds,
encoding the input points into regular grid features (plane or volume) has been
commonly employed in existing approaches. However, these methods typically use
the grid as an index for uniformly scattering point features. Compared with the
irregular point features, the regular grid features may sacrifice some
reconstruction details but improve efficiency. To take full advantage of these
two types of features, we introduce a novel and high-efficiency attention
mechanism between the grid and point features named Point-Grid Transformer
(GridFormer). This mechanism treats the grid as a transfer point connecting the
space and point cloud. Our method maximizes the spatial expressiveness of grid
features and maintains computational efficiency. Furthermore, optimizing
predictions over the entire space could potentially result in blurred
boundaries. To address this issue, we further propose a boundary optimization
strategy incorporating margin binary cross-entropy loss and boundary sampling.
This approach enables us to achieve a more precise representation of the object
structure. Our experiments validate that our method is effective and
outperforms the state-of-the-art approaches under widely used benchmarks by
producing more precise geometry reconstructions. The code is available at
this https URL

通过在正规网格特征和离散点特征之间引入高效的点 - 网格变换机制，以及采用边界优化策略，本文提出的方法有效地提升了三维表面重建的精度和准确性。