When solving a task with limited labelled data, researchers can either use a
general large language model without further update, or use the few examples to
tune a specialised smaller model. When enough labels are available, the
specialised models outperform the general ones on many NLP tasks. In this work,
we aim to investigate how many labelled samples are required for the
specialised models to achieve this superior performance, while taking the
results variance into consideration. Observing the behaviour of prompting,
in-context learning, fine-tuning and instruction-tuning, identifying their
break-even points when increasing number of labelled training samples across
three tasks of varying complexity, we find that the specialised models often
need only few samples ($100-1000$) to be on par or better than the general
ones. At the same time, the amount of required labelled data strongly depends
on the task complexity and results variance.

在本研究中，我们旨在调查专用模型需要多少标记样本才能达到优越性能，同时考虑结果方差。我们通过观察提示、上下文学习、微调和指令调整的行为，确定它们在增加不同复杂度任务的标记训练样本数量时的平衡点，发现专用模型通常只需要少量样本（100-1000）就能达到或超越通用模型。与此同时，所需的标记数据量强烈依赖于任务复杂度和结果方差。

微调、提示、上下文学习和指导微调：我们需要多少标记样本？

Fine-Tuning, Prompting, In-Context Learning and Instruction-Tuning: How  Many Labelled Samples Do We Need?

Federated learning (FL) is a well established method for performing machine
learning tasks over massively distributed data. However in settings where data
is distributed in a non-iid (not independent and identically distributed)
fashion -- as is typical in real world situations -- the joint model produced
by FL suffers in terms of test set accuracy and/or communication costs compared
to training on iid data. We show that learning a single joint model is often
not optimal in the presence of certain types of non-iid data. In this work we
present a modification to FL by introducing a hierarchical clustering step
(FL+HC) to separate clusters of clients by the similarity of their local
updates to the global joint model. Once separated, the clusters are trained
independently and in parallel on specialised models. We present a robust
empirical analysis of the hyperparameters for FL+HC for several iid and non-iid
settings. We show how FL+HC allows model training to converge in fewer
communication rounds (significantly so under some non-iid settings) compared to
FL without clustering. Additionally, FL+HC allows for a greater percentage of
clients to reach a target accuracy compared to standard FL. Finally we make
suggestions for good default hyperparameters to promote superior performing
specialised models without modifying the the underlying federated learning
communication protocol.

本文提出了一种修改的联邦学习 (FL+HC) 方法，通过引入分层聚类步骤将客户端分为相似的集群，然后对这些集群进行独立并行的训练，实现在非独立同分布的数据环境下的模型训练，通过该方法可以使模型的性能、收敛速度、精度都有所提升。