Learning an egocentric action recognition model from video data is
challenging due to distractors (e.g., irrelevant objects) in the background.
Further integrating object information into an action model is hence
beneficial. Existing methods often leverage a generic object detector to
identify and represent the objects in the scene. However, several important
issues remain. Object class annotations of good quality for the target domain
(dataset) are still required for learning good object representation. Besides,
previous methods deeply couple the existing action models and need to retrain
them jointly with object representation, leading to costly and inflexible
integration. To overcome both limitations, we introduce Self-Supervised
Learning Over Sets (SOS), an approach to pre-train a generic Objects In Contact
(OIC) representation model from video object regions detected by an
off-the-shelf hand-object contact detector. Instead of augmenting object
regions individually as in conventional self-supervised learning, we view the
action process as a means of natural data transformations with unique
spatio-temporal continuity and exploit the inherent relationships among
per-video object sets. Extensive experiments on two datasets, EPIC-KITCHENS-100
and EGTEA, show that our OIC significantly boosts the performance of multiple
state-of-the-art video classification models.

采用自助学习技术的 “自我监督学习 over sets”(SOS) 方法，从视频数据的对象区域中提取表示对象的图像特征，结合自然数据转换过程中的时空连续性和对象集内在关系，实现了人物自身操作的行为识别，显著提升了多个最先进的视频分类模型的性能。