In-context learning has become a popular paradigm in natural language
processing. However, its performance can be significantly influenced by the
order of in-context demonstration examples. In this paper, we found that causal
language models (CausalLMs) are more sensitive to this order compared to prefix
language models (PrefixLMs). We attribute this phenomenon to the
auto-regressive attention masks within CausalLMs, which restrict each token
from accessing information from subsequent tokens. This results in different
receptive fields for samples at different positions, thereby leading to
representation disparities across positions. To tackle this challenge, we
introduce an unsupervised fine-tuning method, termed the Information-Augmented
and Consistency-Enhanced approach. This approach utilizes contrastive learning
to align representations of in-context examples across different positions and
introduces a consistency loss to ensure similar representations for inputs with
different permutations. This enhances the model's predictive consistency across
permutations. Experimental results on four benchmarks suggest that our proposed
method can reduce the sensitivity to the order of in-context examples and
exhibit robust generalizability, particularly when demonstrations are sourced
from a pool different from that used in the training phase, or when the number
of in-context examples differs from what is used during training.

我们提出了一种无监督的微调方法，称为信息增强和一致性增强方法，以减少在背景上下文示例的顺序敏感性，并展示出强大的泛化能力，特别是当演示示例来源于与训练阶段不同的池，或者当背景上下文示例数量与训练期间使用的数量不同时。

解决因果语言模型中上下文示例的顺序敏感性问题

Addressing Order Sensitivity of In-Context Demonstration Examples in  Causal Language Models

Most modern approaches for domain adaptive semantic segmentation rely on
continued access to source data during adaptation, which may be infeasible due
to computational or privacy constraints. We focus on source-free domain
adaptation for semantic segmentation, wherein a source model must adapt itself
to a new target domain given only unlabeled target data. We propose
Augmentation Consistency-guided Self-training (AUGCO), a source-free adaptation
algorithm that uses the model's pixel-level predictive consistency across
diverse, automatically generated views of each target image along with model
confidence to identify reliable pixel predictions, and selectively self-trains
on those. AUGCO achieves state-of-the-art results for source-free adaptation on
3 standard benchmarks for semantic segmentation, all within a simple to
implement and fast to converge method.

该研究提出了一种名为 AUGCO 的算法，在无源数据的情况下，利用模型的像素精度在目标图像的多样化自动生成视图下的预测一致性以及模型置信度来识别可靠的像素预测并进行有选择性的自我训练，最终在三种标准的语义分割基准测试中取得了最先进的无源自适应结果，这是一种易于实现和快速收敛的方法。

AUGCO: 无源域自适应语义分割的增强一致性自训练

AUGCO: Augmentation Consistency-guided Self-training for Source-free  Domain Adaptive Semantic Segmentation

Many existing approaches for unsupervised domain adaptation (UDA) focus on
adapting under only data distribution shift and offer limited success under
additional cross-domain label distribution shift. Recent work based on
self-training using target pseudo-labels has shown promise, but on challenging
shifts pseudo-labels may be highly unreliable, and using them for self-training
may cause error accumulation and domain misalignment. We propose Selective
Entropy Optimization via Committee Consistency (SENTRY), a UDA algorithm that
judges the reliability of a target instance based on its predictive consistency
under a committee of random image transformations. Our algorithm then
selectively minimizes predictive entropy to increase confidence on highly
consistent target instances, while maximizing predictive entropy to reduce
confidence on highly inconsistent ones. In combination with pseudo-label based
approximate target class balancing, our approach leads to significant
improvements over the state-of-the-art on 27/31 domain shifts from standard UDA
benchmarks as well as benchmarks designed to stress-test adaptation under label
distribution shift.

提出了一种基于自我训练和预测一致性的无监督域自适应算法（SENTRY），它利用随机图像变换的委员会评估目标实例的可靠性，通过增加高度一致性目标实例的置信度，减少高度不一致实例的置信度来选择性地最小化预测熵和最大化预测熵。该算法结合了基于伪标签的近似目标类平衡方法，在标签分布转移方面具有优异表现。