Reconstructing 3D hand-face interactions with deformations from a single
image is a challenging yet crucial task with broad applications in AR, VR, and
gaming. The challenges stem from self-occlusions during single-view hand-face
interactions, diverse spatial relationships between hands and face, complex
deformations, and the ambiguity of the single-view setting. The first and only
method for hand-face interaction recovery, Decaf, introduces a global fitting
optimization guided by contact and deformation estimation networks trained on
studio-collected data with 3D annotations. However, Decaf suffers from a
time-consuming optimization process and limited generalization capability due
to its reliance on 3D annotations of hand-face interaction data. To address
these issues, we present DICE, the first end-to-end method for
Deformation-aware hand-face Interaction reCovEry from a single image. DICE
estimates the poses of hands and faces, contacts, and deformations
simultaneously using a Transformer-based architecture. It features
disentangling the regression of local deformation fields and global mesh vertex
locations into two network branches, enhancing deformation and contact
estimation for precise and robust hand-face mesh recovery. To improve
generalizability, we propose a weakly-supervised training approach that
augments the training set using in-the-wild images without 3D ground-truth
annotations, employing the depths of 2D keypoints estimated by off-the-shelf
models and adversarial priors of poses for supervision. Our experiments
demonstrate that DICE achieves state-of-the-art performance on a standard
benchmark and in-the-wild data in terms of accuracy and physical plausibility.
Additionally, our method operates at an interactive rate (20 fps) on an Nvidia
4090 GPU, whereas Decaf requires more than 15 seconds for a single image. Our
code will be publicly available upon publication.

DICE 是一种基于 Transformer 的全新的、从单张图像中重建具有变形的手 - 脸互动的方法，其特点是通过将本地变形场和全局网格顶点位置的回归分离到两个网络分支，从而增强精确而稳健的手 - 脸网格恢复中的变形和接触估计。我们的实验表明，在基准测试和野外数据方面，DICE 在准确性和物理合理性方面均达到了最先进的性能水平。此外，我们的方法在 Nvidia 4090 GPU 上以交互速率（20fps）运行，而 Decaf 则需要超过 15 秒处理一张图像。