Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them. In this work, we propose a multi-modal and rich-annotated knowledge repository, OakInk, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: Oak. Given the affordance, we record rich human interactions with 100 selected objects in Oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: Tink. The recorded and transferred hand-object interactions constitute the second knowledge base: Ink. As a result, OakInk contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark OakInk on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of OakInk: intent-based interaction generation and handover generation. Our datasets and source code are publicly available at https://github.com/lixiny/OakInk.

本文提出了一个多模态、丰富注释的知识库OakInk，旨在构建能够对手-物互动进行视觉和认知理解的口袋知识，包括两种知识库：Oak用于记录物体的affordances，Ink用于记录物理世界中的人类互动行为，并使用新颖的方法Tink将这些交互迁移到虚拟现实环境中，本文还介绍了OakInk在姿态估计和握取生成任务中的基准测试以及OakInk的两个实际应用。

OakInk：一个用于理解手与物体交互的大型知识库