Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design. Consequently, the community does not know which techniques perform best or even if they outperform simple baselines. We formulate desiderata for an ideal few-shot NLP benchmark and present FLEX, the first benchmark, public leaderboard, and framework that provides unified, comprehensive measurement for few-shot NLP techniques. FLEX incorporates and introduces new best practices for few-shot evaluation, including measurement of four transfer settings, textual labels for zero-shot evaluation, and a principled approach to benchmark design that optimizes statistical accuracy while keeping evaluation costs accessible to researchers without large compute resources. In addition, we present UniFew, a simple yet strong prompt-based model for few-shot learning which unifies the pretraining and finetuning prompt formats, eschewing complex machinery of recent prompt-based approaches in adapting downstream task formats to language model pretraining objectives. We demonstrate that despite simplicity UniFew achieves results competitive with both popular meta-learning and prompt-based approaches.

提出了FLEX原则，其中最重要的是新的基准设计方法Sample Size Design，可以优化统计精度和评估成本；基于该原则，发布了涵盖四种Few-shot转移设置和零-shot评估的FLEX基准测试，以及UniFew模型，后者能够将预训练和微调提示格式统一起来，并且即使是简单的提示格式，也能够取得与流行的元学习和提示为基础的方法相媲美的结果。

FLEX: 少样本自然语言处理任务的统一评估