This article introduces Lester, a novel method to automatically synthetise
retro-style 2D animations from videos. The method approaches the challenge
mainly as an object segmentation and tracking problem. Video frames are
processed with the Segment Anything Model (SAM) and the resulting masks are
tracked through subsequent frames with DeAOT, a method of hierarchical
propagation for semi-supervised video object segmentation. The geometry of the
masks' contours is simplified with the Douglas-Peucker algorithm. Finally,
facial traits, pixelation and a basic shadow effect can be optionally added.
The results show that the method exhibits an excellent temporal consistency and
can correctly process videos with different poses and appearances, dynamic
shots, partial shots and diverse backgrounds. The proposed method provides a
more simple and deterministic approach than diffusion models based
video-to-video translation pipelines, which suffer from temporal consistency
problems and do not cope well with pixelated and schematic outputs. The method
is also much most practical than techniques based on 3D human pose estimation,
which require custom handcrafted 3D models and are very limited with respect to
the type of scenes they can process.

介绍了一种名为 Lester 的新方法，可以从视频中自动合成复古风格的 2D 动画，主要将这个挑战视为对象分割和跟踪问题，并通过使用 Segment Anything Model (SAM) 对视频帧进行处理，并使用 DeAOT 对生成的掩码进行后续帧的跟踪，最后使用 Douglas-Peucker 算法简化掩码的轮廓，该方法在时间上表现得很一致，并能正确处理不同姿势、外观、动态镜头、部分镜头和多样背景的视频。