Large language models (LLMs) like transformers have impressive in-context
learning (ICL) capabilities; they can generate predictions for new queries
based on input-output sequences in prompts without parameter updates. While
many theories have attempted to explain ICL, they often focus on structured
training data similar to ICL tasks, such as regression. In practice, however,
these models are trained in an unsupervised manner on unstructured text data,
which bears little resemblance to ICL tasks. To this end, we investigate how
ICL emerges from unsupervised training on unstructured data. The key
observation is that ICL can arise simply by modeling co-occurrence information
using classical language models like continuous bag of words (CBOW), which we
theoretically prove and empirically validate. Furthermore, we establish the
necessity of positional information and noise structure to generalize ICL to
unseen data. Finally, we present instances where ICL fails and provide
theoretical explanations; they suggest that the ICL ability of LLMs to identify
certain tasks can be sensitive to the structure of the training data.

通过建模共现信息使用经典语言模型如连续词袋模型，我们理论上证明并实证验证了不监督训练对于上下文学习的产生方式，同时我们提出了一些 ICL 失败的情况并给出了理论解释。

关于共现、位置信息和噪音结构的作用：从无结构数据的训练中产生上下文学习

How In-Context Learning Emerges from Training on Unstructured Data: On  the Role of Co-Occurrence, Positional Information, and Noise Structures

Intelligent Process Automation (IPA) is an emerging technology with a primary
goal to assist the knowledge worker by taking care of repetitive, routine and
low-cognitive tasks. Conversational agents that can interact with users in a
natural language are potential application for IPA systems. Such intelligent
agents can assist the user by answering specific questions and executing
routine tasks that are ordinarily performed in a natural language (i.e.,
customer support). In this work, we tackle a challenge of implementing an IPA
conversational assistant in a real-world industrial setting with a lack of
structured training data. Our proposed system brings two significant benefits:
First, it reduces repetitive and time-consuming activities and, therefore,
allows workers to focus on more intelligent processes. Second, by interacting
with users, it augments the resources with structured and to some extent
labeled training data. We showcase the usage of the latter by re-implementing
several components of our system with Transfer Learning (TL) methods.

本文介绍在缺乏结构化训练数据的真实工业场景中实现智能过程自动化 (IPA) 交谈助手的挑战，以及使用迁移学习方法重新实现自动化系统多个组件，从而为用户提供结构化且在某种程度上标记的培训数据。