The emergence of in-context learning (ICL) is potentially attributed to two
major abilities: task recognition (TR) for recognizing the task from
demonstrations and utilizing pre-trained priors, and task learning (TL) for
learning from demonstrations. However, relationships between the two abilities
and how such relationships affect the emergence of ICL is unclear. In this
paper, we take the first step by examining the pre-training dynamics of the
emergence of ICL. With carefully designed metrics, we find that these two
abilities are, in fact, competitive during pre-training. Moreover, we observe a
strong negative correlation between the competition and ICL performance.
Further analysis of common pre-training factors (i.e., model size, dataset
size, and data curriculum) demonstrates possible ways to manage the
competition. Based on these insights, we propose a simple yet effective method
to better integrate these two abilities for ICL at inference time. Through
adaptive ensemble learning, the performance of ICL can be significantly
boosted, enabling two small models to outperform a larger one with more than
twice the parameters. The code is available at
this https URL

利用预先训练的先验知识，通过任务识别和任务学习两种能力的竞争来促进上下文学习的出现，并提出了一种简单而有效的方法，在推理时间内更好地整合这两种能力。通过自适应集成学习，可以显著提高上下文学习的性能，使两个小模型的性能超过具有两倍参数的大模型。

探究上下文学习的预训练动态：任务识别与任务学习

Investigating the Pre-Training Dynamics of In-Context Learning: Task  Recognition vs. Task Learning

In this paper, we present an Adaptive Ensemble Learning framework that aims
to boost the performance of deep neural networks by intelligently fusing
features through ensemble learning techniques. The proposed framework
integrates ensemble learning strategies with deep learning architectures to
create a more robust and adaptable model capable of handling complex tasks
across various domains. By leveraging intelligent feature fusion methods, the
Adaptive Ensemble Learning framework generates more discriminative and
effective feature representations, leading to improved model performance and
generalization capabilities.
We conducted extensive experiments and evaluations on several benchmark
datasets, including image classification, object detection, natural language
processing, and graph-based learning tasks. The results demonstrate that the
proposed framework consistently outperforms baseline models and traditional
feature fusion techniques, highlighting its effectiveness in enhancing deep
learning models' performance. Furthermore, we provide insights into the impact
of intelligent feature fusion on model performance and discuss the potential
applications of the Adaptive Ensemble Learning framework in real-world
scenarios.
The paper also explores the design and implementation of adaptive ensemble
models, ensemble training strategies, and meta-learning techniques, which
contribute to the framework's versatility and adaptability. In conclusion, the
Adaptive Ensemble Learning framework represents a significant advancement in
the field of feature fusion and ensemble learning for deep neural networks,
with the potential to transform a wide range of applications across multiple
domains.

本文提出了一种自适应集成学习框架，旨在通过智能地融合特征来提高深度神经网络的性能。该框架通过集成学习策略和深度学习架构实现更强大和适应性强的模型，能够处理跨多个领域的复杂任务。通过智能特征融合方法，自适应集成学习框架生成更具辨别性和有效性的特征表示，导致改进了模型性能和泛化能力。在多个基准数据集上进行了广泛的实验和评估，包括图像分类，目标检测，自然语言处理和基于图形的学习任务。结果表明，所提出的框架始终优于基准模型和传统的特征融合技术，在增强深度学习模型的性能方面非常有效。此外，本文还探讨了自适应集成模型的设计和实现、集成训练策略和元学习技术，这些都有助于提高框架的多功能性和适应性。综上所述，自适应集成学习框架代表了特征融合和集成学习在深度神经网络领域的重大进展，具有在跨多个领域中改变广泛应用的潜力。