Reconstructing 3D hand-face interactions with deformations from a single
image is a challenging yet crucial task with broad applications in AR, VR, and
gaming. The challenges stem from self-occlusions during single-view hand-face
interactions, diverse spatial relationships between hands and face, complex
deformations, and the ambiguity of the single-view setting. The first and only
method for hand-face interaction recovery, Decaf, introduces a global fitting
optimization guided by contact and deformation estimation networks trained on
studio-collected data with 3D annotations. However, Decaf suffers from a
time-consuming optimization process and limited generalization capability due
to its reliance on 3D annotations of hand-face interaction data. To address
these issues, we present DICE, the first end-to-end method for
Deformation-aware hand-face Interaction reCovEry from a single image. DICE
estimates the poses of hands and faces, contacts, and deformations
simultaneously using a Transformer-based architecture. It features
disentangling the regression of local deformation fields and global mesh vertex
locations into two network branches, enhancing deformation and contact
estimation for precise and robust hand-face mesh recovery. To improve
generalizability, we propose a weakly-supervised training approach that
augments the training set using in-the-wild images without 3D ground-truth
annotations, employing the depths of 2D keypoints estimated by off-the-shelf
models and adversarial priors of poses for supervision. Our experiments
demonstrate that DICE achieves state-of-the-art performance on a standard
benchmark and in-the-wild data in terms of accuracy and physical plausibility.
Additionally, our method operates at an interactive rate (20 fps) on an Nvidia
4090 GPU, whereas Decaf requires more than 15 seconds for a single image. Our
code will be publicly available upon publication.

DICE 是一种基于 Transformer 的全新的、从单张图像中重建具有变形的手 - 脸互动的方法，其特点是通过将本地变形场和全局网格顶点位置的回归分离到两个网络分支，从而增强精确而稳健的手 - 脸网格恢复中的变形和接触估计。我们的实验表明，在基准测试和野外数据方面，DICE 在准确性和物理合理性方面均达到了最先进的性能水平。此外，我们的方法在 Nvidia 4090 GPU 上以交互速率（20fps）运行，而 Decaf 则需要超过 15 秒处理一张图像。

DICE：从单一图像捕捉手脸互动的端到端变形

DICE: End-to-end Deformation Capture of Hand-Face Interactions from a  Single Image

This paper addresses the complex issue of one-shot face stylization, focusing
on the simultaneous consideration of appearance and structure, where previous
methods have fallen short. We explore deformation-aware face stylization that
diverges from traditional single-image style reference, opting for a real-style
image pair instead. The cornerstone of our method is the utilization of a
self-supervised vision transformer, specifically DINO-ViT, to establish a
robust and consistent facial structure representation across both real and
style domains. Our stylization process begins by adapting the StyleGAN
generator to be deformation-aware through the integration of spatial
transformers (STN). We then introduce two innovative constraints for generator
fine-tuning under the guidance of DINO semantics: i) a directional deformation
loss that regulates directional vectors in DINO space, and ii) a relative
structural consistency constraint based on DINO token self-similarities,
ensuring diverse generation. Additionally, style-mixing is employed to align
the color generation with the reference, minimizing inconsistent
correspondences. This framework delivers enhanced deformability for general
one-shot face stylization, achieving notable efficiency with a fine-tuning
duration of approximately 10 minutes. Extensive qualitative and quantitative
comparisons demonstrate our superiority over state-of-the-art one-shot face
stylization methods. Code is available at
https://github.com/zichongc/DoesFS.

该论文探讨了一次性人脸风格化的复杂问题，通过利用自监督视觉转换器和适应 StyleGAN 生成器进行精细调整，实现了在结构上具有鲁棒性和一致性的面部风格化，达到了显著的效率优势。

通过 DINO 语义引导的可变形一次性人脸风格化

Deformable One-shot Face Stylization via DINO Semantic Guidance

Local feature extraction is a standard approach in computer vision for
tackling important tasks such as image matching and retrieval. The core
assumption of most methods is that images undergo affine transformations,
disregarding more complicated effects such as non-rigid deformations.
Furthermore, incipient works tailored for non-rigid correspondence still rely
on keypoint detectors designed for rigid transformations, hindering performance
due to the limitations of the detector. We propose DALF (Deformation-Aware
Local Features), a novel deformation-aware network for jointly detecting and
describing keypoints, to handle the challenging problem of matching deformable
surfaces. All network components work cooperatively through a feature fusion
approach that enforces the descriptors' distinctiveness and invariance.
Experiments using real deforming objects showcase the superiority of our
method, where it delivers 8% improvement in matching scores compared to the
previous best results. Our approach also enhances the performance of two
real-world applications: deformable object retrieval and non-rigid 3D surface
registration. Code for training, inference, and applications are publicly
available at this https URL

DALF is a new, deformation-aware network for jointly detecting and describing keypoints, which can be used for local feature extraction in computer vision and delivers better performance when dealing with matching deformable surfaces.