Medical students and junior surgeons often rely on senior surgeons and
specialists to answer their questions when learning surgery. However, experts
are often busy with clinical and academic work, and have little time to give
guidance. Meanwhile, existing deep learning (DL)-based surgical Visual Question
Answering (VQA) systems can only provide simple answers without the location of
the answers. In addition, vision-language (ViL) embedding is still a less
explored research in these kinds of tasks. Therefore, a surgical Visual
Question Localized-Answering (VQLA) system would be helpful for medical
students and junior surgeons to learn and understand from recorded surgical
videos. We propose an end-to-end Transformer with Co-Attention gaTed
Vision-Language (CAT-ViL) for VQLA in surgical scenarios, which does not
require feature extraction through detection models. The CAT-ViL embedding
module is designed to fuse heterogeneous features from visual and textual
sources. The fused embedding will feed a standard Data-Efficient Image
Transformer (DeiT) module, before the parallel classifier and detector for
joint prediction. We conduct the experimental validation on public surgical
videos from MICCAI EndoVis Challenge 2017 and 2018. The experimental results
highlight the superior performance and robustness of our proposed model
compared to the state-of-the-art approaches. Ablation studies further prove the
outstanding performance of all the proposed components. The proposed method
provides a promising solution for surgical scene understanding, and opens up a
primary step in the Artificial Intelligence (AI)-based VQLA system for surgical
training. Our code is publicly available.

本研究提出了一种基于端到端 Transformer 与协同注意力门控视觉 - 语言嵌入的外科手术场景中可定位答案的视觉问答系统，该方法不需要通过检测模型进行特征提取，并建立在数据高效图像 Transformer 模块、并行分类器和检测器之上，该方法在公共手术视频实验上的结果表明了与现有技术的优越性。

视觉问答机器人手术中采用共同关注门控视觉语言嵌入的本地化回答

Co-Attention Gated Vision-Language Embedding for Visual Question  Localized-Answering in Robotic Surgery

We consider the problem of omni-supervised object detection, which can use
unlabeled, fully labeled and weakly labeled annotations, such as image tags,
counts, points, etc., for object detection. This is enabled by a unified
architecture, Omni-DETR, based on the recent progress on student-teacher
framework and end-to-end transformer based object detection. Under this unified
architecture, different types of weak labels can be leveraged to generate
accurate pseudo labels, by a bipartite matching based filtering mechanism, for
the model to learn. In the experiments, Omni-DETR has achieved state-of-the-art
results on multiple datasets and settings. And we have found that weak
annotations can help to improve detection performance and a mixture of them can
achieve a better trade-off between annotation cost and accuracy than the
standard complete annotation. These findings could encourage larger object
detection datasets with mixture annotations. The code is available at
this https URL

研究使用统一结构 Omni-DETR 进行 omni-supervised 物体检测，将弱标注转化为准确的伪标注用于学习，取得了多个数据集的最优结果。