Scene graph generation (SGG) of surgical procedures is crucial in enhancing
holistically cognitive intelligence in the operating room (OR). However,
previous works have primarily relied on the multi-stage learning that generates
semantic scene graphs dependent on intermediate processes with pose estimation
and object detection, which may compromise model efficiency and efficacy, also
impose extra annotation burden. In this study, we introduce a novel
single-stage bimodal transformer framework for SGG in the OR, termed
S^2Former-OR, aimed to complementally leverage multi-view 2D scenes and 3D
point clouds for SGG in an end-to-end manner. Concretely, our model embraces a
View-Sync Transfusion scheme to encourage multi-view visual information
interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to
integrate the synergic 2D semantic features into 3D point cloud features.
Moreover, based on the augmented feature, we propose a novel relation-sensitive
transformer decoder that embeds dynamic entity-pair queries and relational
trait priors, which enables the direct prediction of entity-pair relations for
graph generation without intermediate steps. Extensive experiments have
validated the superior SGG performance and lower computational cost of
S^2Former-OR on 4D-OR benchmark, compared with current OR-SGG methods, e.g., 3%
Precision increase and 24.2M reduction in model parameters. We further compared
our method with generic single-stage SGG methods with broader metrics for a
comprehensive evaluation, with consistently better performance achieved. The
code will be made available.

本研究介绍了一种新颖的单阶段双模态变压器框架用于手术室中的场景图生成，名为 S^2Former-OR，旨在以端到端的方式辅助利用多视角的 2D 场景和 3D 点云进行场景图生成。通过引入视图同步转运方案促进多视角视觉信息交互，设计几何视觉凝聚操作将 2D 语义特征与 3D 点云特征进行整合，基于增广特征，提出了一种新颖的关系敏感变压器解码器，嵌入动态实体对查询和关系特征先验，实现了无需中间步骤直接预测实体对关系的图生成。通过广泛的实验证明了 S^2Former-OR 在 4D 手术室基准测试中具有卓越的场景图生成性能和较低的计算成本，与当前 OR-SGG 方法相比，例如提高了 3％的精度并减少了 24.2M 的模型参数。我们进一步将我们的方法与更广泛指标的通用单阶段 SGG 方法进行了全面评估，始终获得更好的性能。代码将提供。