In this work, we focus on the task of procedure planning from instructional
videos with text supervision, where a model aims to predict an action sequence
to transform the initial visual state into the goal visual state. A critical
challenge of this task is the large semantic gap between observed visual states
and unobserved intermediate actions, which is ignored by previous works.
Specifically, this semantic gap refers to that the contents in the observed
visual states are semantically different from the elements of some action text
labels in a procedure. To bridge this semantic gap, we propose a novel
event-guided paradigm, which first infers events from the observed states and
then plans out actions based on both the states and predicted events. Our
inspiration comes from that planning a procedure from an instructional video is
to complete a specific event and a specific event usually involves specific
actions. Based on the proposed paradigm, we contribute an Event-guided
Prompting-based Procedure Planning (E3P) model, which encodes event information
into the sequential modeling process to support procedure planning. To further
consider the strong action associations within each event, our E3P adopts a
mask-and-predict approach for relation mining, incorporating a probabilistic
masking scheme for regularization. Extensive experiments on three datasets
demonstrate the effectiveness of our proposed model.

本研究针对从带有文本监督的指令视频中进行过程规划的任务，提出了一种基于事件引导的范式，通过从观察到的状态中推断事件并基于状态和预测的事件规划行动。在提出的模型中，还采用了遮盖和预测方法进行关系挖掘，通过将事件信息编码到顺序建模过程中来支持过程规划。实验证明了我们提出的模型的有效性。

基于事件引导的通过文本监督的指导视频的流程规划

Event-Guided Procedure Planning from Instructional Videos with Text  Supervision

3D dense captioning is a recently-proposed novel task, where point clouds
contain more geometric information than the 2D counterpart. However, it is also
more challenging due to the higher complexity and wider variety of inter-object
relations contained in point clouds. Existing methods only treat such relations
as by-products of object feature learning in graphs without specifically
encoding them, which leads to sub-optimal results. In this paper, aiming at
improving 3D dense captioning via capturing and utilizing the complex relations
in the 3D scene, we propose MORE, a Multi-Order RElation mining model, to
support generating more descriptive and comprehensive captions. Technically,
our MORE encodes object relations in a progressive manner since complex
relations can be deduced from a limited number of basic ones. We first devise a
novel Spatial Layout Graph Convolution (SLGC), which semantically encodes
several first-order relations as edges of a graph constructed over 3D object
proposals. Next, from the resulting graph, we further extract multiple triplets
which encapsulate basic first-order relations as the basic unit, and construct
several Object-centric Triplet Attention Graphs (OTAG) to infer multi-order
relations for every target object. The updated node features from OTAG are
aggregated and fed into the caption decoder to provide abundant relational
cues, so that captions including diverse relations with context objects can be
generated. Extensive experiments on the Scan2Cap dataset prove the
effectiveness of our proposed MORE and its components, and we also outperform
the current state-of-the-art method. Our code is available at
this https URL.

本文提出 MORE，一种多阶关系挖掘模型，通过逐步建立空间布局图卷积和基于对象中心的三元组注意力图来捕捉和利用三维场景中的复杂关系，以生成更全面和描述性的标题，证明了 MORE 和其组件的有效性，并超过了现有的最新方法。