Talking Face Generation (TFG) aims to reconstruct facial movements to achieve
high natural lip movements from audio and facial features that are under
potential connections. Existing TFG methods have made significant advancements
to produce natural and realistic images. However, most work rarely takes visual
quality into consideration. It is challenging to ensure lip synchronization
while avoiding visual quality degradation in cross-modal generation methods. To
address this issue, we propose a universal High-Definition Teeth Restoration
Network, dubbed HDTR-Net, for arbitrary TFG methods. HDTR-Net can enhance teeth
regions at an extremely fast speed while maintaining synchronization, and
temporal consistency. In particular, we propose a Fine-Grained Feature Fusion
(FGFF) module to effectively capture fine texture feature information around
teeth and surrounding regions, and use these features to fine-grain the feature
map to enhance the clarity of teeth. Extensive experiments show that our method
can be adapted to arbitrary TFG methods without suffering from lip
synchronization and frame coherence. Another advantage of HDTR-Net is its
real-time generation ability. Also under the condition of high-definition
restoration of talking face video synthesis, its inference speed is $300\%$
faster than the current state-of-the-art face restoration based on
super-resolution.

提出了一个名为 HDTR-Net 的通用高清修复网络，用于任意对话生成方法，可以在保持同步和时序一致性的同时，以极快的速度增强牙齿区域。实验证明，该方法适用于任意对话生成方法，同时能够实时生成高清对话视频，比基于超分辨率的当前先进脸部修复方法快 300%。

HDTR-Net: 用于任意讲话人脸生成的实时高清修复牙齿网络

HDTR-Net: A Real-Time High-Definition Teeth Restoration Network for  Arbitrary Talking Face Generation Methods

Transformer, as a strong and flexible architecture for modelling long-range
relations, has been widely explored in vision tasks. However, when used in
video inpainting that requires fine-grained representation, existed method
still suffers from yielding blurry edges in detail due to the hard patch
splitting. Here we aim to tackle this problem by proposing FuseFormer, a
Transformer model designed for video inpainting via fine-grained feature fusion
based on novel Soft Split and Soft Composition operations. The soft split
divides feature map into many patches with given overlapping interval. On the
contrary, the soft composition operates by stitching different patches into a
whole feature map where pixels in overlapping regions are summed up. These two
modules are first used in tokenization before Transformer layers and
de-tokenization after Transformer layers, for effective mapping between tokens
and features. Therefore, sub-patch level information interaction is enabled for
more effective feature propagation between neighboring patches, resulting in
synthesizing vivid content for hole regions in videos. Moreover, in FuseFormer,
we elaborately insert the soft composition and soft split into the feed-forward
network, enabling the 1D linear layers to have the capability of modelling 2D
structure. And, the sub-patch level feature fusion ability is further enhanced.
In both quantitative and qualitative evaluations, our proposed FuseFormer
surpasses state-of-the-art methods. We also conduct detailed analysis to
examine its superiority.

FuseFormer 是一种针对视频修复任务的 Transformer 模型，通过使用细粒度特征融合的新型软分割和软组合操作，实现了更加有效的特征传播和内容合成，优于现有技术方法。