As model context lengths continue to increase, the number of demonstrations
that can be provided in-context approaches the size of entire training
datasets. We study the behavior of in-context learning (ICL) at this extreme
scale on multiple datasets and models. We show that, for many datasets with
large label spaces, performance continues to increase with hundreds or
thousands of demonstrations. We contrast this with example retrieval and
finetuning: example retrieval shows excellent performance at low context
lengths but has diminished gains with more demonstrations; finetuning is more
data hungry than ICL but can sometimes exceed long-context ICL performance with
additional data. We use this ICL setting as a testbed to study several
properties of both in-context learning and long-context models. We show that
long-context ICL is less sensitive to random input shuffling than short-context
ICL, that grouping of same-label examples can negatively impact performance,
and that the performance boosts we see do not arise from cumulative gain from
encoding many examples together. We conclude that although long-context ICL can
be surprisingly effective, most of this gain comes from attending back to
similar examples rather than task learning.

当模型的上下文长度不断增加时，可以提供的演示数量接近整个训练数据集的规模。在多个数据集和模型上，我们研究了在这种极端规模下的上下文学习（ICL）的行为。我们展示了对于许多具有大型标签空间的数据集，性能在数百或数千个演示中仍在增加。与示例检索和微调相比，示例检索在较短的上下文长度下表现出色，但随着更多的演示，增益减弱；微调比 ICL 更依赖数据，但有时可以通过额外数据超过长上下文 ICL 的性能。我们在 ICL 设置中使用这个作为试验平台来研究上下文学习和长上下文模型的几个特性。我们展示了长上下文 ICL 比短上下文 ICL 对于随机输入乱序不太敏感，相同标签示例的分组可能对性能产生负面影响，并且我们观察到的性能提升并非来自一起编码多个示例的累积增益。我们得出结论，尽管长上下文 ICL 可能出人意料地有效，但其中大部分增益来自于重新关注类似示例而非任务学习。

长篇上下文模型的现场学习：深入探索

In-Context Learning with Long-Context Models: An In-Depth Exploration

Domain generalization (DG) is an important problem that learns a model that
can generalize to unseen test domains leveraging one or more source domains,
under the assumption of shared label spaces. However, most DG methods assume
access to abundant source data in the target label space, a requirement that
proves overly stringent for numerous real-world applications, where acquiring
the same label space as the target task is prohibitively expensive. For this
setting, we tackle the multimodal version of the unsupervised domain
generalization (UDG) problem, which uses a large task-agnostic unlabeled source
dataset, such as LAION-2B during finetuning. Our framework does not explicitly
assume any relationship between the source dataset and target task. Instead, it
relies only on the premise that the source dataset can be efficiently searched
in a joint vision-language space. For this multimodal UDG setting, we propose a
novel method to build a small ($<$100K) subset of the source data in three
simple steps: (1) diversified retrieval using label names as queries, (2) rank
pseudo-labeling, and (3) clustering to find representative samples. To
demonstrate the value of studying the multimodal UDG problem, we compare our
results against state-of-the-art source-free DG and zero-shot (ZS) methods on
their respective benchmarks and show up to 10% improvement in accuracy on 20
diverse target datasets. Additionally, our multi-stage dataset construction
method achieves 3% improvement on average over nearest neighbors retrieval.
Code is available: this https URL

该研究论文探讨了领域泛化和无监督领域泛化问题，提出了一种在多模态场景下构建数据集的新方法，并在不同的基准测试上与其他方法进行了比较，取得了较高的准确性。

基于数据中心化方法的无监督领域泛化：来自网络规模多模态数据的检索

A Data Centric Approach for Unsupervised Domain Generalization via  Retrieval from Web Scale Multimodal Data

Given multiple datasets with different label spaces, the goal of this work is
to train a single object detector predicting over the union of all the label
spaces. The practical benefits of such an object detector are obvious and
significant application-relevant categories can be picked and merged form
arbitrary existing datasets. However, naive merging of datasets is not possible
in this case, due to inconsistent object annotations. Consider an object
category like faces that is annotated in one dataset, but is not annotated in
another dataset, although the object itself appears in the latter images. Some
categories, like face here, would thus be considered foreground in one dataset,
but background in another. To address this challenge, we design a framework
which works with such partial annotations, and we exploit a pseudo labeling
approach that we adapt for our specific case. We propose loss functions that
carefully integrate partial but correct annotations with complementary but
noisy pseudo labels. Evaluation in the proposed novel setting requires full
annotation on the test set. We collect the required annotations and define a
new challenging experimental setup for this task based one existing public
datasets. We show improved performances compared to competitive baselines and
appropriate adaptations of existing work.

本文提出了一种框架，通过融合多个数据集的标签空间进行单一目标检测，解决了不同注释数据集之间的矛盾问题，采用伪标签方法处理局部而正确的注释，提出了损失函数将伪标签与噪声控制在一个相对稳定的范围内，并通过实验表明了方法的有效性。