We propose Filtering Inversion (FINV), a learning framework and optimization
process that predicts a renderable 3D object representation from one or few
partial views. FINV addresses the challenge of synthesizing novel views of
objects from partial observations, spanning cases where the object is not
entirely in view, is partially occluded, or is only observed from similar
views. To achieve this, FINV learns shape priors by training a 3D generative
model. At inference, given one or more views of a novel real-world object, FINV
first finds a set of latent codes for the object by inverting the generative
model from multiple initial seeds. Maintaining the set of latent codes, FINV
filters and resamples them after receiving each new observation, akin to
particle filtering. The generator is then finetuned for each latent code on the
available views in order to adapt to novel objects. We show that FINV
successfully synthesizes novel views of real-world objects (e.g., chairs,
tables, and cars), even if the generative prior is trained only on synthetic
objects. The ability to address the sim-to-real problem allows FINV to be used
for object categories without real-world datasets. FINV achieves
state-of-the-art performance on multiple real-world datasets, recovers object
shape and texture from partial and sparse views, is robust to occlusion, and is
able to incrementally improve its representation with more observations.

本文提出了 Filtering Inversion（FINV），采用 3D 生成模型和优化过程的学习框架，是从一个或少量的部分视图预测可渲染的 3D 物体表示，成功地解决了从部分观察中合成新的对象视图的挑战，实验表明 FINV 在多个真实世界数据集上实现了最先进的性能，并能够从部分稀疏视图中恢复对象形状和纹理，并且能够随着观察量的增加而逐步改进其表示。