This paper proposes a novel application system for the generation of
three-dimensional (3D) character animation driven by markerless human body
motion capturing. The entire pipeline of the system consists of five stages: 1)
the capturing of motion data using multiple cameras, 2) detection of the
two-dimensional (2D) human body joints, 3) estimation of the 3D joints, 4)
calculation of bone transformation matrices, and 5) generation of character
animation. The main objective of this study is to generate a 3D skeleton and
animation for 3D characters using multi-view images captured by ordinary
cameras. The computational complexity of the 3D skeleton reconstruction based
on 3D vision has been reduced as needed to achieve frame-by-frame motion
capturing. The experimental results reveal that our system can effectively and
efficiently capture human actions and use them to animate 3D cartoon characters
in real-time.

本文描述了一种基于无标记人体动作捕获的三维角色动画生成系统，采用视角多样的相机捕捉人体动作信息，并通过多种技术手段计算出骨骼变换状况，从而能够高效准确地实现三维骨骼重建与实时动画生成。

基于多视角摄像机的无标记身体运动捕捉技术用于 3D 角色动画制作

Markerless Body Motion Capturing for 3D Character Animation based on Multi-view Cameras

We propose the first approach to automatically and jointly synthesize both
the synchronous 3D conversational body and hand gestures, as well as 3D face
and head animations, of a virtual character from speech input. Our algorithm
uses a CNN architecture that leverages the inherent correlation between facial
expression and hand gestures. Synthesis of conversational body gestures is a
multi-modal problem since many similar gestures can plausibly accompany the
same input speech. To synthesize plausible body gestures in this setting, we
train a Generative Adversarial Network (GAN) based model that measures the
plausibility of the generated sequences of 3D body motion when paired with the
input audio features. We also contribute a new way to create a large corpus of
more than 33 hours of annotated body, hand, and face data from in-the-wild
videos of talking people. To this end, we apply state-of-the-art monocular
approaches for 3D body and hand pose estimation as well as dense 3D face
performance capture to the video corpus. In this way, we can train on orders of
magnitude more data than previous algorithms that resort to complex in-studio
motion capture solutions, and thereby train more expressive synthesis
algorithms. Our experiments and user study show the state-of-the-art quality of
our speech-synthesized full 3D character animations.

利用 CNN 架构及 GAN 模型自动综合逼真的 3D 虚拟人物之肢体手势，面部表情和语音生成，同时创造出三维身体、手部和面部数据，并利用多模态的 3D 姿态估计技术和训练，实现更加自然流畅的合成动画。