Training or finetuning large-scale language models (LLMs) requires
substantial computation resources, motivating recent efforts to explore
parameter-efficient adaptation to downstream tasks. One approach is to treat
these models as black boxes and use forward passes (Inference APIs) to interact
with them. Current research focuses on adapting these black-box models to
downstream tasks using gradient-free prompt optimization, but this often
involves an expensive process of searching task-specific prompts. Therefore, we
are motivated to study black-box language model adaptation without prompt
search. Specifically, we introduce a label-enhanced cross-attention network
called CrossTune, which models the semantic relatedness between the input text
sequence and task-specific label descriptions. Its effectiveness is examined in
the context of few-shot text classification. To improve the generalization of
CrossTune, we utilize ChatGPT to generate additional training data through
in-context learning. A switch mechanism is implemented to exclude low-quality
ChatGPT-generated data. Through extensive experiments on seven benchmark text
classification datasets, we demonstrate that our proposed approach outperforms
the previous state-of-the-art gradient-free black-box tuning method by 5.7% on
average. Even without using ChatGPT-augmented data, CrossTune performs better
or comparably than previous black-box tuning methods, suggesting the
effectiveness of our approach.

通过引入一种称为 CrossTune 的标签增强的交叉注意力网络，该网络在输入文本序列和任务特定标签描述之间建模语义相关性，本研究在少样本文本分类的背景下检验了 CrossTune 的有效性。通过使用 ChatGPT 生成额外的训练数据以提高 CrossTune 的泛化能力，并通过实验验证提出的方法相较于以往梯度自由黑盒调优方法平均提升了 5.7%，即使不使用 ChatGPT 增强数据，CrossTune 的性能也优于或与以往黑盒调优方法相当，这表明了我们方法的有效性。