Animating virtual characters has always been a fundamental research problem
in virtual reality (VR). Facial animations play a crucial role as they
effectively convey emotions and attitudes of virtual humans. However, creating
such facial animations can be challenging, as current methods often involve
utilization of expensive motion capture devices or significant investments of
time and effort from human animators in tuning animation parameters. In this
paper, we propose a holistic solution to automatically animate virtual human
faces. In our solution, a deep learning model was first trained to retarget the
facial expression from input face images to virtual human faces by estimating
the blendshape coefficients. This method offers the flexibility of generating
animations with characters of different appearances and blendshape topologies.
Second, a practical toolkit was developed using Unity 3D, making it compatible
with the most popular VR applications. The toolkit accepts both image and video
as input to animate the target virtual human faces and enables users to
manipulate the animation results. Furthermore, inspired by the spirit of
Human-in-the-loop (HITL), we leveraged user feedback to further improve the
performance of the model and toolkit, thereby increasing the customization
properties to suit user preferences. The whole solution, for which we will make
the code public, has the potential to accelerate the generation of facial
animations for use in VR applications.

提出了一种全面的解决方案，通过深度学习模型将面部表情从输入面部图像转移到虚拟人脸，实现虚拟人物面部动画的自动化。使用 Unity 3D 开发了实用工具包，可接受图像和视频作为输入，以动画形式呈现目标虚拟人脸，并允许用户对动画结果进行操作。通过用户反馈不断改进模型和工具包的性能，从而增加了定制化属性以适应用户偏好。整个解决方案有助于加速用于虚拟现实应用的面部动画的生成。

自带角色：定制角色自动面部动画生成的整体解决方案

Bring Your Own Character: A Holistic Solution for Automatic Facial  Animation Generation of Customized Characters

The consistent mapping from poems to paintings is essential for the research
and restoration of traditional Chinese gardens. But the lack of firsthand
ma-terial is a great challenge to the reconstruction work. In this paper, we
pro-pose a method to generate garden paintings based on text descriptions using
deep learning method. Our image-text pair dataset consists of more than one
thousand Ming Dynasty Garden paintings and their inscriptions and post-scripts.
A latent text-to-image diffusion model learns the mapping from de-scriptive
texts to garden paintings of the Ming Dynasty, and then the text description of
Jichang Garden guides the model to generate new garden paintings. The cosine
similarity between the guide text and the generated image is the evaluation
criterion for the generated images. Our dataset is used to fine-tune the
pre-trained diffusion model using Low-Rank Adapta-tion of Large Language Models
(LoRA). We also transformed the generated images into a panorama and created a
free-roam scene in Unity 3D. Our post-trained model is capable of generating
garden images in the style of Ming Dynasty landscape paintings based on textual
descriptions. The gener-ated images are compatible with three-dimensional
presentation in Unity 3D.

利用深度学习方法，基于文本描述生成了模仿明代园林画风格的园林画。