In this paper, we present TOSS, which introduces text to the task of novel
view synthesis (NVS) from just a single RGB image. While Zero-1-to-3 has
demonstrated impressive zero-shot open-set NVS capability, it treats NVS as a
pure image-to-image translation problem. This approach suffers from the
challengingly under-constrained nature of single-view NVS: the process lacks
means of explicit user control and often results in implausible NVS
generations. To address this limitation, TOSS uses text as high-level semantic
information to constrain the NVS solution space. TOSS fine-tunes text-to-image
Stable Diffusion pre-trained on large-scale text-image pairs and introduces
modules specifically tailored to image and camera pose conditioning, as well as
dedicated training for pose correctness and preservation of fine details.
Comprehensive experiments are conducted with results showing that our proposed
TOSS outperforms Zero-1-to-3 with more plausible, controllable and
multiview-consistent NVS results. We further support these results with
comprehensive ablations that underscore the effectiveness and potential of the
introduced semantic guidance and architecture design.

本文介绍了 TOSS，它利用文本信息将单一 RGB 图像转化为创新视角合成（NVS）的任务。TOSS 通过使用文本作为高级语义信息来约束 NVS 的解空间，利用特定于图像和相机姿态的模块进行训练，并引入了针对姿态正确性和精细细节保留的专门训练。实验结果表明，我们提出的 TOSS 在可信度、可控性和多视图一致性方面优于 Zero-1-to-3，并通过全面的剖析验证了引入的语义指导和架构设计的有效性和潜力。