Building quality machine learning models for natural language understanding (NLU) tasks relies heavily on labeled data. Weak supervision has been shown to provide valuable supervision when large amount of labeled data is unavailable or expensive to obtain. Existing works studying weak supervision for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels. To date a benchmark for NLU with real world weak supervision signals for a collection of NLU tasks is still not available. In this paper, we propose such a benchmark, named WALNUT, to advocate and facilitate research on weak supervision for NLU. WALNUT consists of NLU tasks with different types, including both document-level prediction tasks and token-level prediction tasks and for each task contains weak labels generated by multiple real-world weak sources. We conduct baseline evaluations on the benchmark to systematically test the value of weak supervision for NLU tasks, with various weak supervision methods and model architectures. We demonstrate the benefits of weak supervision for low-resource NLU tasks and expect WALNUT to stimulate further research on methodologies to best leverage weak supervision. The benchmark and code for baselines will be publicly available at aka.ms/walnut_benchmark.

本文提出了一个名为 WALNUT 的基准测试（semi-WeAkly supervised Learning for Natural language Understanding Testbed）来为 NLU 上的弱监督方法研究提供便利。WALNUT 包含了各种不同类型的 NLU 任务，包括文档级别和令牌级别的预测任务，并与一小部分清洁标签一起生成多个真实世界弱标记来源生成弱标签。我们在 WALNUT 上进行基线评估，以系统地评估各种弱监督方法和模型架构的有效性。

WALNUT：半弱监督自然语言理解基准