In-context learning (ICL) has revolutionized the capabilities of transformer
models in NLP. In our project, we extend the understanding of the mechanisms
underpinning ICL by exploring whether transformers can learn from sequential,
non-textual function class data distributions. We introduce a novel sliding
window sequential function class and employ toy-sized transformers with a GPT-2
architecture to conduct our experiments. Our analysis indicates that these
models can indeed leverage ICL when trained on non-textual sequential function
classes. Additionally, our experiments with randomized y-label sequences
highlights that transformers retain some ICL capabilities even when the label
associations are obfuscated. We provide evidence that transformers can reason
with and understand sequentiality encoded within function classes, as reflected
by the effective learning of our proposed tasks. Our results also show that the
performance deteriorated with increasing randomness in the labels, though not
to the extent one might expect, implying a potential robustness of learned
sequentiality against label noise. Future research may want to look into how
previous explanations of transformers, such as induction heads and task
vectors, relate to sequentiality in ICL in these toy examples. Our
investigation lays the groundwork for further research into how transformers
process and perceive sequential data.

我们的研究扩展了前文所提到的上下文学习在转换模型中的应用，通过探索转换模型是否能够从序列、非文本函数类数据分布中学习。我们引入了一种新颖的滑动窗口序列函数类，并使用了具有 GPT-2 架构的规模较小的转换模型进行实验。我们的分析表明，这些模型确实可以在训练非文本序列函数类时利用上下文学习。此外，我们通过随机的 y 标签序列实验发现，即使标签关联被混淆，转换模型仍保留一定的上下文学习能力。我们提供了证据表明，转换模型可以理解并推理出函数类中所编码的序列性，从而在我们提出的任务中有效学习。我们的结果还显示，性能随着标签的随机性增加而下降，尽管不及预期的程度，暗示了所学习的序列性对标签噪声具有潜在的鲁棒性。未来的研究可以探讨转换模型中的解释性头部和任务向量与上下文学习中的序列性之间的关系。我们的研究为进一步研究转换模型如何处理和感知序列数据奠定了基础。

Transformer 模型能否在上下文中学习顺序功能类别？

Can Transformers Learn Sequential Function Classes In Context?

Multi-label classification is an approach which allows a datapoint to be
labelled with more than one class at the same time. A common but trivial
approach is to train individual binary classifiers per label, but the
performance can be improved by considering associations within the labels. Like
with any machine learning algorithm, hyperparameter tuning is important to
train a good multi-label classifier model. The task of selecting the best
hyperparameter settings for an algorithm is an optimisation problem. Very
limited work has been done on automatic hyperparameter tuning and AutoML in the
multi-label domain. This paper attempts to fill this gap by proposing a neural
network algorithm, CascadeML, to train multi-label neural network based on
cascade neural networks. This method requires minimal or no hyperparameter
tuning and also considers pairwise label associations. The cascade algorithm
grows the network architecture incrementally in a two phase process as it
learns the weights using adaptive first order gradient algorithm, therefore
omitting the requirement of preselecting the number of hidden layers, nodes and
the learning rate. The method was tested on 10 multi-label datasets and
compared with other multi-label classification algorithms. Results show that
CascadeML performs very well without hyperparameter tuning.

本论文在多标签领域提出一种神经网络算法，CascadeML，用于基于级联神经网络的多标签神经网络的训练，无需或最小化超参数调整，并考虑标签之间的关联，同时能够获得很好的性能。