In this paper, we focus on the problem of integrating Energy-based Models
(EBM) as guiding priors for motion optimization. EBMs are a set of neural
networks that can represent expressive probability density distributions in
terms of a Gibbs distribution parameterized by a suitable energy function. Due
to their implicit nature, they can easily be integrated as optimization factors
or as initial sampling distributions in the motion optimization problem, making
them good candidates to integrate data-driven priors in the motion optimization
problem. In this work, we present a set of required modeling and algorithmic
choices to adapt EBMs into motion optimization. We investigate the benefit of
including additional regularizers in the learning of the EBMs to use them with
gradient-based optimizers and we present a set of EBM architectures to learn
generalizable distributions for manipulation tasks. We present multiple cases
in which the EBM could be integrated for motion optimization and evaluate the
performance of learned EBMs as guiding priors for both simulated and real robot
experiments.

本研究聚焦于将能量模型（EBM）作为动作优化的指导先验进行集成的问题，并研究了在学习 EBMs 时包括额外的正则化器的益处以及多种情况下 EBM 能够集成到运动优化中的表现及性能评估。

学习运动优化的隐式先验

Learning Implicit Priors for Motion Optimization

Capturing challenging human motions is critical for numerous applications,
but it suffers from complex motion patterns and severe self-occlusion under the
monocular setting. In this paper, we propose ChallenCap -- a template-based
approach to capture challenging 3D human motions using a single RGB camera in a
novel learning-and-optimization framework, with the aid of multi-modal
references. We propose a hybrid motion inference stage with a generation
network, which utilizes a temporal encoder-decoder to extract the motion
details from the pair-wise sparse-view reference, as well as a motion
discriminator to utilize the unpaired marker-based references to extract
specific challenging motion characteristics in a data-driven manner. We further
adopt a robust motion optimization stage to increase the tracking accuracy, by
jointly utilizing the learned motion details from the supervised multi-modal
references as well as the reliable motion hints from the input image reference.
Extensive experiments on our new challenging motion dataset demonstrate the
effectiveness and robustness of our approach to capture challenging human
motions.

本文提出了 ChallenCap—— 一种基于模板的方法，利用单个 RGB 相机在新颖的学习和优化框架下，结合多模态参考信息，捕捉复杂的人体三维运动。通过使用两个稀疏视角参考点和无配对基于标记的参考点，结合一个生成网络和运动鉴别器，提出了一个混合运动推断阶段，以数据驱动的方式提取特定的挑战性运动特征。同时，通过运动优化阶段结合监督多模态参考提高跟踪精度。在新的运动挑战数据集上的实验证明了我们的方法的有效性和稳健性。

ChallenCap: 使用多模参考实现具有挑战性的人类表现的单目 3D 捕捉

ChallenCap: Monocular 3D Capture of Challenging Human Performances using  Multi-Modal References

This paper proposes a real-time dynamic scene reconstruction method capable
of reproducing the motion, geometry, and segmentation simultaneously given live
depth stream from a single RGB-D camera. Our approach fuses geometry frame by
frame and uses a segmentation-enhanced node graph structure to drive the
deformation of geometry in registration step. A two-level node motion
optimization is proposed. The optimization space of node motions and the range
of physically-plausible deformations are largely reduced by taking advantage of
the articulated motion prior, which is solved by an efficient node graph
segmentation method. Compared to previous fusion-based dynamic scene
reconstruction methods, our experiments show robust and improved reconstruction
results for tangential and occluded motions.

本文提出了一种实时动态场景重构方法，能够在给定单个 RGB-D 相机的实时深度流情况下同时再现运动、几何和分割。我们的方法通过融合几何逐帧进行处理，并使用分割增强的节点图结构来驱动几何变形。我们提出了一种二级节点运动优化方法，并通过利用关节运动先验知识来大大减少节点运动的优化空间和物理合理的变形范围。与之前基于融合的动态场景重构方法相比，我们的实验表明，我们的方法对于切向运动和遮挡的运动都具有鲁棒性和改进的重建结果。