This paper aims to advance the progress of physical world interactive scene
reconstruction by extending the interactive object reconstruction from single
object level to complex scene level. To this end, we first construct one
simulated and one real scene-level physical interaction dataset containing 28
scenes with multiple interactive objects per scene. Furthermore, to accurately
model the interactive motions of multiple objects in complex scenes, we propose
LiveScene, the first scene-level language-embedded interactive neural radiance
field that efficiently reconstructs and controls multiple interactive objects
in complex scenes. LiveScene introduces an efficient factorization that
decomposes the interactive scene into multiple local deformable fields to
separately reconstruct individual interactive objects, achieving the first
accurate and independent control on multiple interactive objects in a complex
scene. Moreover, we introduce an interaction-aware language embedding method
that generates varying language embeddings to localize individual interactive
objects under different interactive states, enabling arbitrary control of
interactive objects using natural language. Finally, we evaluate LiveScene on
the constructed datasets OminiSim and InterReal with various simulated and
real-world complex scenes. Extensive experiment results demonstrate that the
proposed approach achieves SOTA novel view synthesis and language grounding
performance, surpassing existing methods by +9.89, +1.30, and +1.99 in PSNR on
CoNeRF Synthetic, OminiSim #chanllenging, and InterReal #chanllenging datasets,
and +65.12 of mIOU on OminiSim, respectively. Project page:
\href{https://livescenes.github.io}{this https URL}.

通过将单一对象级别的交互式物体重建扩展到复杂场景级别，本研究旨在推进物理世界交互式场景重建的进展。为了准确模拟复杂场景中多个物体的交互运动，我们提出了 LiveScene，即第一个嵌入场景级语言交互式神经辐射场，它能够高效重建和控制复杂场景中的多个交互式物体。通过将交互场景分解为多个局部可变形场，分别重建各个交互式对象，实现了对复杂场景中多个交互式物体的首个准确且独立的控制。此外，我们引入了一种交互感知的语言嵌入方法，通过生成不同交互状态下的语言嵌入来定位各个交互式对象，实现使用自然语言对交互式物体的任意控制。最后，我们利用构建的 OminiSim 和 InterReal 数据集对 LiveScene 进行评估，并在各种模拟和真实世界的复杂场景上展示了出色的新视角合成和语言注释性能，分别在 CoNeRF Synthetic、OminiSim #chanllenging 和 InterReal #chanllenging 数据集上的 PSNR 指标上超过现有方法 9.89、1.30 和 1.99，以及在 OminiSim 数据集上的 mIOU 指标上超过 65.12。

LiveScene: 语言嵌入交互式辐射场用于物理场景渲染和控制

LiveScene: Language Embedding Interactive Radiance Fields for Physical  Scene Rendering and Control

The task of retrieving video content relevant to natural language queries
plays a critical role in effectively handling internet-scale datasets. Most of
the existing methods for this caption-to-video retrieval problem do not fully
exploit cross-modal cues present in video. Furthermore, they aggregate
per-frame visual features with limited or no temporal information. In this
paper, we present a multi-modal transformer to jointly encode the different
modalities in video, which allows each of them to attend to the others. The
transformer architecture is also leveraged to encode and model the temporal
information. On the natural language side, we investigate the best practices to
jointly optimize the language embedding together with the multi-modal
transformer. This novel framework allows us to establish state-of-the-art
results for video retrieval on three datasets. More details are available at
this http URL

本文提出了一种基于多模态 transformer 架构的视频检索方法，该方法能够充分利用视频中的跨模态线索，并融合先前的时间信息。我们还研究了联合优化语言嵌入和多模态 transformer 的最佳实践。该方法在三个数据集上取得了最新的视频检索结果。