Domain-specific text classification faces the challenge of scarce labeled
data due to the high cost of manual labeling. Prompt-learning, known for its
efficiency in few-shot scenarios, is proposed as an alternative to traditional
fine-tuning methods. And besides, although large language models (LLMs) have
gained prominence, small language models (SLMs, with under 1B parameters) offer
significant customizability, adaptability, and cost-effectiveness for
domain-specific tasks, given industry constraints. In this study, we
investigate the potential of SLMs combined with prompt-learning paradigm for
domain-specific text classification, specifically within customer-agent
interactions in retail. Our evaluations show that, in few-shot settings when
prompt-based model fine-tuning is possible, T5-base, a typical SLM with 220M
parameters, achieve approximately 75% accuracy with limited labeled data (up to
15% of full data), which shows great potentials of SLMs with prompt-learning.
Based on this, We further validate the effectiveness of active few-shot
sampling and the ensemble strategy in the prompt-learning pipeline that
contribute to a remarkable performance gain. Besides, in zero-shot settings
with a fixed model, we underscore a pivotal observation that, although the
GPT-3.5-turbo equipped with around 154B parameters garners an accuracy of
55.16%, the power of well designed prompts becomes evident when the
FLAN-T5-large, a model with a mere 0.5% of GPT-3.5-turbo's parameters, achieves
an accuracy exceeding 31% with the optimized prompt, a leap from its sub-18%
performance with an unoptimized one. Our findings underscore the promise of
prompt-learning in classification tasks with SLMs, emphasizing the benefits of
active few-shot sampling, and ensemble strategies in few-shot settings, and the
importance of prompt engineering in zero-shot settings.

在本研究中，我们调查了小语言模型（具有不到 10 亿参数）与 prompt-learning 范例相结合，在零样本和少样本场景下针对零售业中客户 - 代理商互动的领域特定文本分类的潜力。我们的评估结果显示，在少样本设置下进行基于提示的模型微调时，220M 参数的典型小语言模型 T5-base 可以在有限的标记数据（高达全数据的 15%）上实现约 75% 的准确性，显示了小语言模型与 prompt-learning 的巨大潜力。基于此，我们进一步验证了主动少样本抽样和 prompt-learning 流程中的集成策略对显著性能提升的有效性。此外，在固定模型的零样本设置中，我们强调了一个关键的观察结果，即尽管具有约 1540B 参数的 GPT-3.5-turbo 可以达到 55.16% 的准确性，但当仅有 0.5% 参数的 FLAN-T5-large 使用经过优化的提示时，其准确性超过 31%，相比使用未经优化提示的准确性提升了近 13%。我们的发现强调了使用小语言模型进行 prompt-learning 的分类任务中的潜力，强调了主动少样本抽样和集成策略在少样本设置中的好处，并强调了零样本设置中提示工程的重要性。