Human-Object Interaction (HOI) detection lies at the core of action
understanding. Besides 2D information such as human/object appearance and
locations, 3D pose is also usually utilized in HOI learning since its
view-independence. However, rough 3D body joints just carry sparse body
information and are not sufficient to understand complex interactions. Thus, we
need detailed 3D body shape to go further. Meanwhile, the interacted object in
3D is also not fully studied in HOI learning. In light of these, we propose a
detailed 2D-3D joint representation learning method. First, we utilize the
single-view human body capture method to obtain detailed 3D body, face and hand
shapes. Next, we estimate the 3D object location and size with reference to the
2D human-object spatial configuration and object category priors. Finally, a
joint learning framework and cross-modal consistency tasks are proposed to
learn the joint HOI representation. To better evaluate the 2D ambiguity
processing capacity of models, we propose a new benchmark named Ambiguous-HOI
consisting of hard ambiguous images. Extensive experiments in large-scale HOI
benchmark and Ambiguous-HOI show impressive effectiveness of our method. Code
and data are available at this https URL

本论文提出了一种详细的 2D-3D 联合表示学习方法，能够使计算机更好地理解人物和物体之间的交互行为，效果在大规模的 HOI 基准和 Ambiguous-HOI 数据集上表现良好。