Human activities can be learned from video. With effective modeling it is
possible to discover not only the action labels but also the temporal
structures of the activities such as the progression of the sub-activities.
Automatically recognizing such structure from raw video signal is a new
capability that promises authentic modeling and successful recognition of
human-object interactions. Toward this goal, we introduce Asynchronous-Sparse
Interaction Graph Networks (ASSIGN), a recurrent graph network that is able to
automatically detect the structure of interaction events associated with
entities in a video scene. ASSIGN pioneers learning of autonomous behavior of
video entities including their dynamic structure and their interaction with the
coexisting neighbors. Entities' lives in our model are asynchronous to those of
others therefore more flexible in adaptation to complex scenarios. Their
interactions are sparse in time hence more faithful to the true underlying
nature and more robust in inference and learning. ASSIGN is tested on
human-object interaction recognition and shows superior performance in
segmenting and labeling of human sub-activities and object affordances from raw
videos. The native ability for discovering temporal structures of the model
also eliminates the dependence on external segmentation that was previously
mandatory for this task.

通过介绍一种新的循环图网络架构 Asynchronous-Sparse Interaction Graph Networks，将其实施于人物与物品交互行为识别中，通过发掘视频片段中实体间的行为事件结构，自动实现视频实体与其邻居间的动态结构交互的学习，从而消除了对原处理单元的外部区分和先验知识的依赖，使其标记和分割人的子行为和物体切入更为准确，具备更强的鲁棒性。

在视频中学习异步和稀疏的人 - 物互动

Learning Asynchronous and Sparse Human-Object Interaction in Videos

Getting a better understanding of user behavior is important for advancing
information retrieval systems. Existing work focuses on modeling and predicting
single interaction events, such as clicks. In this paper, we for the first time
focus on modeling and predicting sequences of interaction events. And in
particular, sequences of clicks.
We formulate the problem of click sequence prediction and propose a click
sequence model (CSM) that aims to predict the order in which a user will
interact with search engine results. CSM is based on a neural network that
follows the encoder-decoder architecture. The encoder computes contextual
embeddings of the results. The decoder predicts the sequence of positions of
the clicked results. It uses an attention mechanism to extract necessary
information about the results at each timestep. We optimize the parameters of
CSM by maximizing the likelihood of observed click sequences.
We test the effectiveness of CSM on three new tasks: (i) predicting click
sequences, (ii) predicting the number of clicks, and (iii) predicting whether
or not a user will interact with the results in the order these results are
presented on a search engine result page (SERP). Also, we show that CSM
achieves state-of-the-art results on a standard click prediction task, where
the goal is to predict an unordered set of results a user will click on.

该研究提出了一种基于神经网络的点击序列模型，可用于预测用户与搜索引擎结果的交互行为，并在多个预测任务中取得了最先进的结果。