Videos from edited media like movies are a useful, yet under-explored source
of information. The rich variety of appearance and interactions between humans
depicted over a large temporal context in these films could be a valuable
source of data. However, the richness of data comes at the expense of
fundamental challenges such as abrupt shot changes and close up shots of actors
with heavy truncation, which limits the applicability of existing human 3D
understanding methods. In this paper, we address these limitations with an
insight that while shot changes of the same scene incur a discontinuity between
frames, the 3D structure of the scene still changes smoothly. This allows us to
handle frames before and after the shot change as multi-view signal that
provide strong cues to recover the 3D state of the actors. We propose a
multi-shot optimization framework, which leads to improved 3D reconstruction
and mining of long sequences with pseudo ground truth 3D human mesh. We show
that the resulting data is beneficial in the training of various human mesh
recovery models: for single image, we achieve improved robustness; for video we
propose a pure transformer-based temporal encoder, which can naturally handle
missing observations due to shot changes in the input frames. We demonstrate
the importance of the insight and proposed models through extensive
experiments. The tools we develop open the door to processing and analyzing in
3D content from a large library of edited media, which could be helpful for
many downstream applications. Project page:
this https URL

本文提出了使用多角度辅助信号进行多镜头优化框架，以改善人类 3D 重建的方法，并在大量实验中证明了模型的重要性