Despite the availability of computer-aided simulators and recorded videos of
surgical procedures, junior residents still heavily rely on experts to answer
their queries. However, expert surgeons are often overloaded with clinical and
academic workloads and limit their time in answering. For this purpose, we
develop a surgical question-answering system to facilitate robot-assisted
surgical scene and activity understanding from recorded videos. Most of the
existing VQA methods require an object detector and regions based feature
extractor to extract visual features and fuse them with the embedded text of
the question for answer generation. However, (1) surgical object detection
model is scarce due to smaller datasets and lack of bounding box annotation;
(2) current fusion strategy of heterogeneous modalities like text and image is
naive; (3) the localized answering is missing, which is crucial in complex
surgical scenarios. In this paper, we propose Visual Question
Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific
surgical area during the answer prediction. To deal with the fusion of the
heterogeneous modalities, we design gated vision-language embedding (GVLE) to
build input patches for the Language Vision Transformer (LViT) to predict the
answer. To get localization, we add the detection head in parallel with the
prediction head of the LViT. We also integrate GIoU loss to boost localization
performance by preserving the accuracy of the question-answering model. We
annotate two datasets of VQLA by utilizing publicly available surgical videos
from MICCAI challenges EndoVis-17 and 18. Our validation results suggest that
Surgical-VQLA can better understand the surgical scene and localize the
specific area related to the question-answering. GVLE presents an efficient
language-vision embedding technique by showing superior performance over the
existing benchmarks.

本文提出一种在机器人手术场景中定位特定手术区域的视觉问答系统，使用门控视觉语言嵌入（GVLE）和语言视觉变压器（LViT）来进行异构模态的融合和回答的预测，GVLE 在实现语言 - 视觉嵌入方面表现出优异的性能，并与现有基准相比速度更快，并通过添加检测头来实现本地化的回答预测。