In this work, we propose a novel method to supervise 3D Gaussian Splatting
(3DGS) scenes using optical tactile sensors. Optical tactile sensors have
become widespread in their use in robotics for manipulation and object
representation; however, raw optical tactile sensor data is unsuitable to
directly supervise a 3DGS scene. Our representation leverages a Gaussian
Process Implicit Surface to implicitly represent the object, combining many
touches into a unified representation with uncertainty. We merge this model
with a monocular depth estimation network, which is aligned in a two stage
process, coarsely aligning with a depth camera and then finely adjusting to
match our touch data. For every training image, our method produces a
corresponding fused depth and uncertainty map. Utilizing this additional
information, we propose a new loss function, variance weighted depth supervised
loss, for training the 3DGS scene model. We leverage the DenseTact optical
tactile sensor and RealSense RGB-D camera to show that combining touch and
vision in this manner leads to quantitatively and qualitatively better results
than vision or touch alone in a few-view scene syntheses on opaque as well as
on reflective and transparent objects. Please see our project page at
this http URL

使用光学触觉传感器监督 3D 高斯扩散（3DGS）场景，利用高斯过程隐式曲面将触摸数据和视觉深度估计网络相结合，提出一种新的损失函数（方差加权深度监督损失）训练 3DGS 场景模型，实验证明该方法在几景合成上能够比单独使用视觉或触觉更好地从不透明、反射及透明物体中获取定量和定性的结果。

Touch-GS：视觉 - 触觉监督的 3D 高斯点云渲染

Touch-GS: Visual-Tactile Supervised 3D Gaussian Splatting

This paper deals with the scarcity of data for training optical flow
networks, highlighting the limitations of existing sources such as labeled
synthetic datasets or unlabeled real videos. Specifically, we introduce a
framework to generate accurate ground-truth optical flow annotations quickly
and in large amounts from any readily available single real picture. Given an
image, we use an off-the-shelf monocular depth estimation network to build a
plausible point cloud for the observed scene. Then, we virtually move the
camera in the reconstructed environment with known motion vectors and rotation
angles, allowing us to synthesize both a novel view and the corresponding
optical flow field connecting each pixel in the input image to the one in the
new frame. When trained with our data, state-of-the-art optical flow networks
achieve superior generalization to unseen real data compared to the same models
trained either on annotated synthetic datasets or unlabeled videos, and better
specialization if combined with synthetic images.

本文介绍了一种从真实图片中快速生成大量准确的光流标注的框架，该框架利用单目深度估计网络构建可能的点云，通过虚拟相机运动合成新视角和对应的光流场，实现对现有数据的有效利用，并提高了视觉光流网络在真实场景中的泛化和特化性能。