Recent advancements in Surgical Visual Question Answering (Surgical-VQA) and
related region grounding have shown great promise for robotic and medical
applications, addressing the critical need for automated methods in
personalized surgical mentorship. However, existing models primarily provide
simple structured answers and struggle with complex scenarios due to their
limited capability in recognizing long-range dependencies and aligning
multimodal information. In this paper, we introduce Surgical-LVLM, a novel
personalized large vision-language model tailored for complex surgical
scenarios. Leveraging the pre-trained large vision-language model and
specialized Visual Perception LoRA (VP-LoRA) blocks, our model excels in
understanding complex visual-language tasks within surgical contexts. In
addressing the visual grounding task, we propose the Token-Interaction (TIT)
module, which strengthens the interaction between the grounding module and the
language responses of the Large Visual Language Model (LVLM) after projecting
them into the latent space. We demonstrate the effectiveness of Surgical-LVLM
on several benchmarks, including EndoVis-17-VQLA, EndoVis-18-VQLA, and a newly
introduced EndoVis Conversations dataset, which sets new performance standards.
Our work contributes to advancing the field of automated surgical mentorship by
providing a context-aware solution.

介绍了 Surgical-LVLM，这是一种专为复杂手术场景定制的个性化大型视觉 - 语言模型，通过引入 VP-LoRA 模块和 TIT 模块，能够在手术背景下理解复杂的视觉 - 语言任务，并在多个基准测试中展现了出色的性能，为自动化手术辅导领域做出了贡献。