Vanilla pixel-level classifiers for semantic segmentation are based on a
certain paradigm, involving the inner product of fixed prototypes obtained from
the training set and pixel features in the test image. This approach, however,
encounters significant limitations, i.e., feature deviation in the semantic
domain and information loss in the spatial domain. The former struggles with
large intra-class variance among pixel features from different images, while
the latter fails to utilize the structured information of semantic objects
effectively. This leads to blurred mask boundaries as well as a deficiency of
fine-grained recognition capability. In this paper, we propose a novel Semantic
and Spatial Adaptive (SSA) classifier to address the above challenges.
Specifically, we employ the coarse masks obtained from the fixed prototypes as
a guide to adjust the fixed prototype towards the center of the semantic and
spatial domains in the test image. The adapted prototypes in semantic and
spatial domains are then simultaneously considered to accomplish classification
decisions. In addition, we propose an online multi-domain distillation learning
strategy to improve the adaption process. Experimental results on three
publicly available benchmarks show that the proposed SSA significantly improves
the segmentation performance of the baseline models with only a minimal
increase in computational cost. Code is available at
this https URL

本文提出了一种新颖的语义和空间自适应（SSA）分类器，通过使用从固定原型获得的粗略掩码作为指导，在测试图像的语义和空间领域调整固定原型，从而同时考虑语义和空间领域中的自适应原型以完成分类决策，文章实验证明 SSA 显著提高了基线模型的分割性能，且计算成本仅略有增加。

语义分割的语义空间自适应像素级分类器

Semantic and Spatial Adaptive Pixel-level Classifier for Semantic  Segmentation

We analyze the operation of transformer language adapters, which are small
modules trained on top of a frozen language model to adapt its predictions to
new target languages. We show that adapted predictions mostly evolve in the
source language the model was trained on, while the target language becomes
pronounced only in the very last layers of the model. Moreover, the adaptation
process is gradual and distributed across layers, where it is possible to skip
small groups of adapters without decreasing adaptation performance. Last, we
show that adapters operate on top of the model's frozen representation space
while largely preserving its structure, rather than on an 'isolated' subspace.
Our findings provide a deeper view into the adaptation process of language
models to new languages, showcasing the constraints imposed on it by the
underlying model and introduces practical implications to enhance its
efficiency.

通过分析变压器语言适配器的操作，我们表明适应某个新的目标语言的预测主要在模型训练时所使用的源语言上进行了演化，而目标语言仅在模型的最后几层中变得明显。此外，适应过程是渐进的，并分布在各个层次上，可以跳过少量适配器而不降低适应性能。最后，我们发现适配器在模型的冻结表示空间上操作，同时在很大程度上保留其结构，而不是在 “独立” 的子空间上操作。我们的研究结果深入了解了语言模型对新语言的适应过程，展示了底层模型所施加的约束，并引入了增强其效率的实际应用意义。

Transformer 语言适配器的隐藏空间

The Hidden Space of Transformer Language Adapters

The paradigm of pre-training and fine-tuning has laid the foundation for
deploying deep learning models. However, most fine-tuning methods are designed
to meet a specific resource budget. Recently, considering diverse deployment
scenarios with various resource budgets, stitchable neural network (SN-Net) is
introduced to quickly obtain numerous new networks (stitches) from the
pre-trained models (anchors) in a model family via model stitching. Although
promising, SN-Net confronts new challenges when adapting it to new target
domains, including huge memory and storage requirements and a long and
sub-optimal multistage adaptation process. In this work, we present a novel
framework, Efficient Stitchable Task Adaptation (ESTA), to efficiently produce
a palette of fine-tuned models that adhere to diverse resource constraints.
Specifically, we first tailor parameter-efficient fine-tuning to share low-rank
updates among the stitches while maintaining independent bias terms. In this
way, we largely reduce fine-tuning memory burdens and mitigate the interference
among stitches that arises in task adaptation. Furthermore, we streamline a
simple yet effective one-stage deployment pipeline, which estimates the
important stitches to deploy with training-time gradient statistics. By
assigning higher sampling probabilities to important stitches, we also get a
boosted Pareto frontier. Extensive experiments on 25 downstream visual
recognition tasks demonstrate that our ESTA is capable of generating stitches
with smooth accuracy-efficiency trade-offs and surpasses the direct SN-Net
adaptation by remarkable margins with significantly lower training time and
fewer trainable parameters. Furthermore, we demonstrate the flexibility and
scalability of our ESTA framework by stitching LLMs from LLaMA family,
obtaining chatbot stitches of assorted sizes.

通过有效的可缝合任务适应 (ESTA) 框架，我们能够以低延迟、高效率的方式生成一系列满足多样化资源限制的经过优化的模型。