Modern operating room is becoming increasingly complex, requiring innovative
intra-operative support systems. While the focus of surgical data science has
largely been on video analysis, integrating surgical computer vision with
language capabilities is emerging as a necessity. Our work aims to advance
Visual Question Answering (VQA) in the surgical context with scene graph
knowledge, addressing two main challenges in the current surgical VQA systems:
removing question-condition bias in the surgical VQA dataset and incorporating
scene-aware reasoning in the surgical VQA model design. First, we propose a
Surgical Scene Graph-based dataset, SSG-QA, generated by employing segmentation
and detection models on publicly available datasets. We build surgical scene
graphs using spatial and action information of instruments and anatomies. These
graphs are fed into a question engine, generating diverse QA pairs. Our SSG-QA
dataset provides a more complex, diverse, geometrically grounded, unbiased, and
surgical action-oriented dataset compared to existing surgical VQA datasets. We
then propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight
Scene-embedded Interaction Module (SIM), which integrates geometric scene
knowledge in the VQA model design by employing cross-attention between the
textual and the scene features. Our comprehensive analysis of the SSG-QA
dataset shows that SSG-QA-Net outperforms existing methods across different
question types and complexities. We highlight that the primary limitation in
the current surgical VQA systems is the lack of scene knowledge to answer
complex queries. We present a novel surgical VQA dataset and model and show
that results can be significantly improved by incorporating geometric scene
features in the VQA model design. The source code and the dataset will be made
publicly available at: this https URL

通过使用场景图知识解决目前手术 VQA 系统中的问题条件偏见和融入场景感知推理两个挑战，我们提出了一种基于手术场景图的数据集 SSG-QA 和一种新颖的手术 VQA 模型 SSG-QA-Net，展示了通过将几何场景特征融入 VQA 模型设计中能够显著提高结果。