We study the problem of creating a character model that can be controlled in
real time from a single image of an anime character. A solution to this problem
would greatly reduce the cost of creating avatars, computer games, and other
interactive applications.
Talking Head Anime 3 (THA3) is an open source project that attempts to
directly addresses the problem. It takes as input (1) an image of an anime
character's upper body and (2) a 45-dimensional pose vector and outputs a new
image of the same character taking the specified pose. The range of possible
movements is expressive enough for personal avatars and certain types of game
characters. However, the system is too slow to generate animations in real time
on common PCs, and its image quality can be improved.
In this paper, we improve THA3 in two ways. First, we propose new
architectures for constituent networks that rotate the character's head and
body based on U-Nets with attention that are widely used in modern generative
models. The new architectures consistently yield better image quality than the
THA3 baseline. Nevertheless, they also make the whole system much slower: it
takes up to 150 milliseconds to generate a frame. Second, we propose a
technique to distill the system into a small network (less than 2 MB) that can
generate 512x512 animation frames in real time (under 30 FPS) using consumer
gaming GPUs while keeping the image quality close to that of the full system.
This improvement makes the whole system practical for real-time applications.

我们研究了使用单一动漫角色图像实时控制角色模型的问题，并通过提出新的构成网络架构和技术改进现有系统，使其在保持图像质量的同时，能够在实时应用中生成高质量动画帧。

单图像生成说话人动漫的改进模型及其蒸馏

Talking Head(?) Anime from a Single Image 4: Improved Model and Its  Distillation

We propose PAniC-3D, a system to reconstruct stylized 3D character heads
directly from illustrated (p)ortraits of (ani)me (c)haracters. Our anime-style
domain poses unique challenges to single-view reconstruction; compared to
natural images of human heads, character portrait illustrations have hair and
accessories with more complex and diverse geometry, and are shaded with
non-photorealistic contour lines. In addition, there is a lack of both 3D model
and portrait illustration data suitable to train and evaluate this ambiguous
stylized reconstruction task. Facing these challenges, our proposed PAniC-3D
architecture crosses the illustration-to-3D domain gap with a line-filling
model, and represents sophisticated geometries with a volumetric radiance
field. We train our system with two large new datasets (11.2k Vroid 3D models,
1k Vtuber portrait illustrations), and evaluate on a novel AnimeRecon benchmark
of illustration-to-3D pairs. PAniC-3D significantly outperforms baseline
methods, and provides data to establish the task of stylized reconstruction
from portrait illustrations.

该论文提出了一种名为 PAniC-3D 的系统，可通过插图直接重建动漫角色头部的立体化模型。它通过使用线条填充技术与体积辐射场技术来跨越插画到三维重建的领域鸿沟，并利用两个新的大型数据集和一个基准测试数据集进行训练和评估。研究结果表明，PAniC-3D 系统的表现明显优于传统的基线方法，为建立从插图到三维重建的任务提供了良好的数据基础。