Understanding how humans would behave during hand-object interaction is vital
for applications in service robot manipulation and extended reality. To achieve
this, some recent works have been proposed to simultaneously predict hand
trajectories and object affordances on human egocentric videos. They are
regarded as the representation of future hand-object interactions, indicating
potential human motion and motivation. However, the existing approaches mostly
adopt the autoregressive paradigm for unidirectional prediction, which lacks
mutual constraints within the holistic future sequence, and accumulates errors
along the time axis. Meanwhile, these works basically overlook the effect of
camera egomotion on first-person view predictions. To address these
limitations, we propose a novel diffusion-based interaction prediction method,
namely Diff-IP2D, to forecast future hand trajectories and object affordances
concurrently in an iterative non-autoregressive manner. We transform the
sequential 2D images into latent feature space and design a denoising diffusion
model to predict future latent interaction features conditioned on past ones.
Motion features are further integrated into the conditional denoising process
to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate
interaction prediction. The experimental results show that our method
significantly outperforms the state-of-the-art baselines on both the
off-the-shelf metrics and our proposed new evaluation protocol. This highlights
the efficacy of leveraging a generative paradigm for 2D hand-object interaction
prediction. The code of Diff-IP2D will be released at
this https URL

我们提出了一种新的基于扩散的交互预测方法 Diff-IP2D，以非自回归的迭代方式同时预测未来的手轨迹和物体可供性，在预测过程中考虑了相机使用者的动态特征，实验证明我们的方法在 2D 手物交互预测方面优于现有的基线方法。

基于扩散的自视角视频中手物交互预测

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on  Egocentric Videos

We propose robust methods for estimating camera egomotion in noisy,
real-world monocular image sequences in the general case of unknown observer
rotation and translation with two views and a small baseline. This is a
difficult problem because of the nonconvex cost function of the perspective
camera motion equation and because of non-Gaussian noise arising from noisy
optical flow estimates and scene non-rigidity. To address this problem, we
introduce the expected residual likelihood method (ERL), which estimates
confidence weights for noisy optical flow data using likelihood distributions
of the residuals of the flow field under a range of counterfactual model
parameters. We show that ERL is effective at identifying outliers and
recovering appropriate confidence weights in many settings. We compare ERL to a
novel formulation of the perspective camera motion equation using a lifted
kernel, a recently proposed optimization framework for joint parameter and
confidence weight estimation with good empirical properties. We incorporate
these strategies into a motion estimation pipeline that avoids falling into
local minima. We find that ERL outperforms the lifted kernel method and
baseline monocular egomotion estimation strategies on the challenging KITTI
dataset, while adding almost no runtime cost over baseline egomotion methods.

使用期望残差似然方法（ERL）对噪声、光流估计和场景非刚性引起的非高斯噪声进行建模，有效地识别异常值并在许多设置中恢复恰当的置信度权重，从而超过基线单目自我运动估计策略。