Fine-tuning pretrained language models (LMs) without making any architectural
changes has become a norm for learning various language downstream tasks.
However, for non-language downstream tasks, a common practice is to employ
task-specific designs for input, output layers, and loss functions. For
instance, it is possible to fine-tune an LM into an MNIST classifier by
replacing the word embedding layer with an image patch embedding layer, the
word token output layer with a 10-way output layer, and the word prediction
loss with a 10-way classification loss, respectively. A natural question
arises: Can LM fine-tuning solve non-language downstream tasks without changing
the model architecture or loss function? To answer this, we propose
Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations
by conducting an extensive empirical study on a suite of non-language
classification and regression tasks. LIFT does not make any changes to the
model architecture or loss function, and it solely relies on the natural
language interface, enabling "no-code machine learning with LMs." We find that
LIFT performs comparably well across a wide range of low-dimensional
classification and regression tasks, matching the performances of the best
baselines in many cases, especially for the classification tasks. We also
report experimental results on the fundamental properties of LIFT, including
inductive bias, robustness, and sample complexity. We also analyze the effect
of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g.,
context-aware learning via appropriate prompting, calibrated predictions, data
generation, and two-stage fine-tuning. Our code is available at
this https URL.

本研究通过语言接口微调（LIFT）来解决非语言下游任务，而不更改模型架构或损失函数，并进行了全面的实证研究，发现在低维分类和回归任务上，LIFT 的表现与最佳基线相当，并具有归纳偏置，鲁棒性和样本复杂性。

LIFT: 面向非语言机器学习任务的语言接口微调

LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

We investigate the capability of a transformer pretrained on natural language
to generalize to other modalities with minimal finetuning -- in particular,
without finetuning of the self-attention and feedforward layers of the residual
blocks. We consider such a model, which we call a Frozen Pretrained Transformer
(FPT), and study finetuning it on a variety of sequence classification tasks
spanning numerical computation, vision, and protein fold prediction. In
contrast to prior works which investigate finetuning on the same modality as
the pretraining dataset, we show that pretraining on natural language can
improve performance and compute efficiency on non-language downstream tasks.
Additionally, we perform an analysis of the architecture, comparing the
performance of a random initialized transformer to a random LSTM. Combining the
two insights, we find language-pretrained transformers can obtain strong
performance on a variety of non-language tasks.

本研究探究了预先在自然语言处理上训练的 transformer 模型是否可以在最小化调整下进行泛化以适用于其他模式，并研究其在数字计算、视觉和蛋白质折叠预测等序列分类任务上的优化性能。结果表明，预训练模型对非语言下游任务的性能和计算效率都有积极作用。