Multimodal synthetic data generation is crucial in domains such as autonomous
driving, robotics, augmented/virtual reality, and retail. We propose a novel
approach, GenMM, for jointly editing RGB videos and LiDAR scans by inserting
temporally and geometrically consistent 3D objects. Our method uses a reference
image and 3D bounding boxes to seamlessly insert and blend new objects into
target videos. We inpaint the 2D Regions of Interest (consistent with 3D boxes)
using a diffusion-based video inpainting model. We then compute semantic
boundaries of the object and estimate it's surface depth using state-of-the-art
semantic segmentation and monocular depth estimation techniques. Subsequently,
we employ a geometry-based optimization algorithm to recover the 3D shape of
the object's surface, ensuring it fits precisely within the 3D bounding box.
Finally, LiDAR rays intersecting with the new object surface are updated to
reflect consistent depths with its geometry. Our experiments demonstrate the
effectiveness of GenMM in inserting various 3D objects across video and LiDAR
modalities.

多模态合成数据生成对于自动驾驶、机器人、增强 / 虚拟现实和零售等领域至关重要。我们提出了一种新颖的方法 GenMM，通过插入时间和几何一致的 3D 对象来联合编辑 RGB 视频和 LiDAR 扫描。我们的方法使用参考图像和三维边界框将新对象无缝地插入和融入目标视频。我们使用基于扩散的视频修复模型修复与三维框一致的 2D 感兴趣区域。然后，我们使用最先进的语义分割和单目深度估计技术计算对象的语义边界并估计其表面深度。随后，我们使用基于几何的优化算法恢复对象表面的 3D 形状，确保其准确适应三维边界框。最后，与新对象表面相交的 LiDAR 光线被更新以反映与其几何形状一致的深度。我们的实验证明了 GenMM 在视频和 LiDAR 模态中插入各种 3D 对象的有效性。