Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a
source language to make predictions in another language, often with a
performance loss. To alleviate this, additional improvements can be achieved
through subsequent adaptation using examples in the target language. In this
paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer
in the classification task by introducing In-Context Cross-lingual Transfer
(IC-XLT). The novel concept involves training a model to learn from context
examples and subsequently adapting it during inference to a target language by
prepending a One-Shot context demonstration in that language. Our results show
that IC-XLT successfully leverages target-language examples to improve the
cross-lingual capabilities of the evaluated mT5 model, outperforming
prompt-based models in the Zero and Few-shot scenarios adapted through
fine-tuning. Moreover, we show that when source-language data is limited, the
fine-tuning framework employed for IC-XLT performs comparably to prompt-based
fine-tuning with significantly more training data in the source language.

通过在分类任务中介绍上下文一次性跨语言转换（IC-XLT），可以通过训练模型学习上下文示例，然后在推断时通过在目标语言中预置一次性上下文演示来适应目标语言，提高评估的 mT5 模型的跨语言能力，超过基于提示的模型在经过微调的零和少样本情景中的表现。此外，当源语言数据有限时，IC-XLT 的微调框架与源语言中具有显著更多训练数据的基于提示的微调具有相当的性能。

通过上下文一次性演示实现适应性跨语言文本分类

Adaptive Cross-lingual Text Classification through In-Context One-Shot  Demonstrations

Continual table semantic parsing aims to train a parser on a sequence of
tasks, where each task requires the parser to translate natural language into
SQL based on task-specific tables but only offers limited training examples.
Conventional methods tend to suffer from overfitting with limited supervision,
as well as catastrophic forgetting due to parameter updates. Despite recent
advancements that partially alleviate these issues through semi-supervised data
augmentation and retention of a few past examples, the performance is still
limited by the volume of unsupervised data and stored examples. To overcome
these challenges, this paper introduces a novel method integrating
\textit{parameter-efficient fine-tuning} (PEFT) and \textit{in-context tuning}
(ICT) for training a continual table semantic parser. Initially, we present a
task-adaptive PEFT framework capable of fully circumventing catastrophic
forgetting, which is achieved by freezing the pre-trained model backbone and
fine-tuning small-scale prompts. Building on this, we propose a teacher-student
framework-based solution. The teacher addresses the few-shot problem using ICT,
which procures contextual information by demonstrating a few training examples.
In turn, the student leverages the proposed PEFT framework to learn from the
teacher's output distribution, and subsequently compresses and saves the
contextual information to the prompts, eliminating the need to store any
training examples. Experimental evaluations on two benchmarks affirm the
superiority of our method over prevalent few-shot and continual learning
baselines across various metrics.

本文介绍了一个用于训练连续表语义解析器的新方法，该方法整合了参数高效微调（PEFT）和上下文微调（ICT），通过冻结预训练模型骨干和微调小规模提示来完全避免灾难性遗忘，并且通过教师 - 学生框架解决了少样本学习问题，通过 ICT 获得上下文信息来演示几个训练样本，然后学生利用 PEFT 框架从教师的输出分布中学习，并将上下文信息压缩保存到提示中，消除了存储任何训练样本的需要。在两个基准测试中进行的实验评估确认了我们方法在各种度量标准上相对于常见的少样本和连续学习基准的优越性。

参数化上下文：释放参数高效微调和上下文调整在连续表格语义解析中的能力

Parameterizing Context: Unleashing the Power of Parameter-Efficient  Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing

The goal of meta-learning is to learn to adapt to a new task with only a few
labeled examples. To tackle this problem in NLP, we propose $\textit{in-context
tuning}$, which recasts adaptation and prediction as a simple sequence
prediction problem: to form the input sequence, we concatenate the task
instruction, the labeled examples, and the target input to predict; to
meta-train the model to learn from in-context examples, we fine-tune a
pre-trained language model (LM) to predict the target label from the input
sequences on a collection of tasks.
We benchmark our method on two collections of text classification tasks: LAMA
and BinaryClfs. Compared to first-order MAML which adapts the model with
gradient descent, our method better leverages the inductive bias of LMs to
perform pattern matching, and outperforms MAML by an absolute $6\%$ AUC ROC
score on BinaryClfs, with increasing advantage w.r.t. model size. Compared to
non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning
directly learns to learn from in-context examples. On BinaryClfs, in-context
tuning improves the average AUC-ROC score by an absolute $10\%$, and reduces
the variance with respect to example ordering by 6x and example choices by 2x.

通过将调整和预测转化为简单的序列预测问题，我们提出了 'in-context tuning' 方式来解决 NLP 中的元学习问题，并在两个文本分类任务组合上对其进行了基准测试。在所有模型尺寸上，相对于梯度下降调整模型的一阶 MAML，我们的方法更好地利用了 LM 的归纳偏差，在 BinaryClfs 上绝对 AUC ROC 评分提高了 6％。与不调优的 'in-context learning' 相比较，'in-context tuning' 直接通过元训练从 'in-context' 中学习。在 BinaryClfs 上，'in-context tuning' 将平均 AUC-ROC 分数提高了 10％，并且减少了关于示例顺序的方差 6 倍，示例选择 2 倍。