In this paper, we address the challenge of image resolution variation for the
Segment Anything Model (SAM). SAM, known for its zero-shot generalizability,
exhibits a performance degradation when faced with datasets with varying image
sizes. Previous approaches tend to resize the image to a fixed size or adopt
structure modifications, hindering the preservation of SAM's rich prior
knowledge. Besides, such task-specific tuning necessitates a complete
retraining of the model, which is cost-expensive and unacceptable for
deployment in the downstream tasks. In this paper, we reformulate this issue as
a length extrapolation problem, where token sequence length varies while
maintaining a consistent patch size for images of different sizes. To this end,
we propose Scalable Bias-Mode Attention Mask (BA-SAM) to enhance SAM's
adaptability to varying image resolutions while eliminating the need for
structure modifications. Firstly, we introduce a new scaling factor to ensure
consistent magnitude in the attention layer's dot product values when the token
sequence length changes. Secondly, we present a bias-mode attention mask that
allows each token to prioritize neighboring information, mitigating the impact
of untrained distant information. Our BA-SAM demonstrates efficacy in two
scenarios: zero-shot and fine-tuning. Extensive evaluation on diverse datasets,
including DIS5K, DUTS, ISIC, COD10K, and COCO, reveals its ability to
significantly mitigate performance degradation in the zero-shot setting and
achieve state-of-the-art performance with minimal fine-tuning. Furthermore, we
propose a generalized model and benchmark, showcasing BA-SAM's generalizability
across all four datasets simultaneously.

本文提出了 Scalable Bias-Mode Attention Mask（BA-SAM）解决图像分辨率变化对 Segment Anything Model (SAM) 的影响，并实现了零样本泛化能力。通过引入新的缩放因子和偏置模式注意力掩码，BA-SAM 在不改变结构的情况下，增强了模型对不同图像分辨率的适应性，同时在零样本学习和微调中取得了显著的性能提升。

BA-SAM：可扩展的偏置注意力掩码用于分割任何模型

BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model

Transformer has taken the natural language processing (NLP) field by storm
since birth, owing to its superior ability to model complex dependencies in
sequences. Despite the great success of pretrained language models (PLMs) based
on Transformer across almost all NLP tasks, they all suffer from a preset
length limit and thus can hardly extend this success to longer sequences beyond
seen data, namely the length extrapolation problem. Length extrapolation has
aroused great interest among researchers, as it is the core feature of human
language capacity. To enhance length extrapolation of Transformers, a plethora
of methods have been proposed, mostly focusing on extrapolatable position
encodings. In this article, we provide an organized and systematical review of
these research efforts in a unified notation from a position encoding
perspective, aiming to enable the reader to gain a deep understanding of
existing methods and provide stimuli for future research.

该研究论文对基于 Transformer 的预训练语言模型和长度外推问题进行了系统性的回顾，聚焦于可外推位置编码方法，旨在帮助读者深入理解已有方法并为未来研究提供启示。