Generative AI offers a simple, prompt-based alternative to fine-tuning
smaller BERT-style LLMs for text classification tasks. This promises to
eliminate the need for manually labeled training data and task-specific model
training. However, it remains an open question whether tools like ChatGPT can
deliver on this promise. In this paper, we show that smaller, fine-tuned LLMs
(still) consistently and significantly outperform larger, zero-shot prompted
models in text classification. We compare three major generative AI models
(ChatGPT with GPT-3.5/GPT-4 and Claude Opus) with several fine-tuned LLMs
across a diverse set of classification tasks (sentiment, approval/disapproval,
emotions, party positions) and text categories (news, tweets, speeches). We
find that fine-tuning with application-specific training data achieves superior
performance in all cases. To make this approach more accessible to a broader
audience, we provide an easy-to-use toolkit alongside this paper. Our toolkit,
accompanied by non-technical step-by-step guidance, enables users to select and
fine-tune BERT-like LLMs for any classification task with minimal technical and
computational effort.

通过比较生成型 AI 模型和经过细调的 LLMs，在文本分类任务中，利用应用特定训练数据进行细调的模型表现优异，从而验证了生成型 AI 模型在兑现其承诺方面存在问题。