Surgical triplet recognition is an essential building block to enable
next-generation context-aware operating rooms. The goal is to identify the
combinations of instruments, verbs, and targets presented in surgical video
frames. In this paper, we propose DiffTriplet, a new generative framework for
surgical triplet recognition employing the diffusion model, which predicts
surgical triplets via iterative denoising. To handle the challenge of triplet
association, two unique designs are proposed in our diffusion framework, i.e.,
association learning and association guidance. During training, we optimize the
model in the joint space of triplets and individual components to capture the
dependencies among them. At inference, we integrate association constraints
into each update of the iterative denoising process, which refines the triplet
prediction using the information of individual components. Experiments on the
CholecT45 and CholecT50 datasets show the superiority of the proposed method in
achieving a new state-of-the-art performance for surgical triplet recognition.
Our codes will be released.

在本文中，我们提出了 DiffTriplet，一种利用扩散模型的新的生成性手术三元组识别框架，通过迭代去噪来预测手术三元组。我们的 Diffusion 框架提出了两种独特的设计来解决三元组关联的挑战，即关联学习和关联引导。实验证明，我们的方法在手术三元组识别方面取得了卓越的性能，达到了最新的最先进水平。

通过扩散模型进行手术三元组识别

Surgical Triplet Recognition via Diffusion Model

CNNs and Self attention have achieved great success in multimedia
applications for dynamic association learning of self-attention and convolution
in image restoration. However, CNNs have at least two shortcomings: 1) limited
receptive field; 2) static weight of sliding window at inference, unable to
cope with the content diversity.In view of the advantages and disadvantages of
CNNs and Self attention, this paper proposes an association learning method to
utilize the advantages and suppress their shortcomings, so as to achieve
high-quality and efficient inpainting. We regard rain distribution reflects the
degradation location and degree, in addition to the rain distribution
prediction. Thus, we propose to refine background textures with the predicted
degradation prior in an association learning manner. As a result, we accomplish
image deraining by associating rain streak removal and background recovery,
where an image deraining network and a background recovery network are designed
for two subtasks. The key part of association learning is a novel multi-input
attention module. It generates the degradation prior and produces the
degradation mask according to the predicted rainy distribution. Benefited from
the global correlation calculation of SA, MAM can extract the informative
complementary components from the rainy input with the degradation mask, and
then help accurate texture restoration. Meanwhile, SA tends to aggregate
feature maps with self-attention importance, but convolution diversifies them
to focus on the local textures. A hybrid fusion network involves one residual
Transformer branch and one encoder-decoder branch. The former takes a few
learnable tokens as input and stacks multi-head attention and feed-forward
networks to encode global features of the image. The latter, conversely,
leverages the multi-scale encoder-decoder to represent contexture knowledge.

本文提出了一种关联学习方法，利用 CNN 和 self attention 的优点并抑制它们的缺点，实现高质量和高效的图像修复。该方法通过关联雨痕去除和背景恢复来完成图像去雨，其中设计了图像去雨网络和背景恢复网络作为两个子任务，并引入了一种新颖的多输入 attention 模块用于关联学习。

图像恢复中自注意力和卷积的动态关联学习

Dynamic Association Learning of Self-Attention and Convolution in Image  Restoration

We present a neural framework for learning associations between interrelated
groups of words such as the ones found in Subject-Verb-Object (SVO) structures.
Our model induces a joint function-specific word vector space, where vectors of
e.g. plausible SVO compositions lie close together. The model retains
information about word group membership even in the joint space, and can
thereby effectively be applied to a number of tasks reasoning over the SVO
structure. We show the robustness and versatility of the proposed framework by
reporting state-of-the-art results on the tasks of estimating selectional
preference and event similarity. The results indicate that the combinations of
representations learned with our task-independent model outperform
task-specific architectures from prior work, while reducing the number of
parameters by up to 95%.

我们提出了一个用于学习关联性的神经框架，该框架可应用于主谓宾结构等词组的词向量空间的学习。结果表明，我们的模型在选择偏好和事件相似性等任务上均具有鲁棒性和多功能性，并减少了高达 95％的参数数量。