Attention-based transformer models have achieved remarkable progress in
multi-modal tasks, such as visual question answering. The explainability of
attention-based methods has recently attracted wide interest as it can explain
the inner changes of attention tokens by accumulating relevancy across
attention layers. Current methods simply update relevancy by equally
accumulating the token relevancy before and after the attention processes.
However, the importance of token values is usually different during relevance
accumulation. In this paper, we propose a weighted relevancy strategy, which
takes the importance of token values into consideration, to reduce distortion
when equally accumulating relevance. To evaluate our method, we propose a
unified CLIP-based two-stage model, named CLIPmapper, to process
Vision-and-Language tasks through CLIP encoder and a following mapper.
CLIPmapper consists of self-attention, cross-attention, single-modality, and
cross-modality attention, thus it is more suitable for evaluating our generic
explainability method. Extensive perturbation tests on visual question
answering and image captioning validate that our explainability method
outperforms existing methods.

我们提出了一种加权相关性策略，考虑了令牌值的重要性，以减少在平等累积相关性时的失真。通过 CLIP 编码器和后续的映射器，我们提出了统一的基于 CLIP 的两阶段模型，命名为 CLIPmapper，用于处理视觉和语言任务，通过自注意、交叉注意、单模态和交叉模态注意，这使得我们的通用可解释性方法更加合适。对视觉问答和图像字幕进行的广泛扰动测试证实了我们的可解释性方法优于现有的方法。

通过加权相关累加实现通用注意力模型的解释性

Generic Attention-model Explainability by Weighted Relevance  Accumulation

Detecting and segmenting object instances is a common task in biomedical
applications. Examples range from detecting lesions on functional magnetic
resonance images, to the detection of tumours in histopathological images and
extracting quantitative single-cell information from microscopy imagery, where
cell segmentation is a major bottleneck. Attention-based transformers are
state-of-the-art in a range of deep learning fields. They have recently been
proposed for segmentation tasks where they are beginning to outperforming other
methods. We present a novel attention-based cell detection transformer
(Cell-DETR) for direct end-to-end instance segmentation. While the segmentation
performance is on par with a state-of-the-art instance segmentation method,
Cell-DETR is simpler and faster. We showcase the method's contribution in a the
typical use case of segmenting yeast in microstructured environments, commonly
employed in systems or synthetic biology. For the specific use case, the
proposed method surpasses the state-of-the-art tools for semantic segmentation
and additionally predicts the individual object instances. The fast and
accurate instance segmentation performance increases the experimental
information yield for a posteriori data processing and makes online monitoring
of experiments and closed-loop optimal experimental design feasible.

本文介绍了一种基于注意力机制的细胞检测变换器 (Cell-DETR), 该方法可直接进行端到端实例分割，具有快速准确的实现分割性能，并可提高后验数据处理的实验信息输出，实现在线监测实验和封闭回路最优实验设计等功能。