Human actions often induce changes of object states such as "cutting an
apple", "cleaning shoes" or "pouring coffee". In this paper, we seek to
temporally localize object states (e.g. "empty" and "full" cup) together with
the corresponding state-modifying actions ("pouring coffee") in long uncurated
videos with minimal supervision. The contributions of this work are threefold.
First, we develop a self-supervised model for jointly learning state-modifying
actions together with the corresponding object states from an uncurated set of
videos from the Internet. The model is self-supervised by the causal ordering
signal, i.e. initial object state $\rightarrow$ manipulating action
$\rightarrow$ end state. Second, to cope with noisy uncurated training data,
our model incorporates a noise adaptive weighting module supervised by a small
number of annotated still images, that allows to efficiently filter out
irrelevant videos during training. Third, we collect a new dataset with more
than 2600 hours of video and 34 thousand changes of object states, and manually
annotate a part of this data to validate our approach. Our results demonstrate
substantial improvements over prior work in both action and object
state-recognition in video.

本文提出一种自我监督的模型，能够在长视频中联合学习目标物体的状态（例如，空杯子和满杯子）及其相应的状态变化行为（如 “倒咖啡”），并针对噪声数据和未筛选的视频数据分别采用噪声自适应加权模块和新的视频数据集进行监督训练，并在目标动作和物体状态识别方面都实现了相当大的进步。