The recently proposed capability-based NLP testing allows model developers to test the functional capabilities of NLP models, revealing functional failures that cannot be detected by the traditional heldout mechanism. However, existing work on capability-based testing requires extensive manual efforts and domain expertise in creating the test cases. In this paper, we investigate a low-cost approach for the test case generation by leveraging the GPT-3 engine. We further propose to use a classifier to remove the invalid outputs from GPT-3 and expand the outputs into templates to generate more test cases. Our experiments show that TestAug has three advantages over the existing work on behavioral testing: (1) TestAug can find more bugs than existing work; (2) The test cases in TestAug are more diverse; and (3) TestAug largely saves the manual efforts in creating the test suites. The code and data for TestAug can be found at our project website (https://guanqun-yang.github.io/testaug/) and GitHub (https://github.com/guanqun-yang/testaug).

本文介绍了一种基于GPT-3引擎生成测试用例的低成本方法，并提出使用分类器将输出扩展为模板生成更多测试用例。测试结果表明TestAug比现有方法具有更多的优势：（1）TestAug可以发现更多的错误；（2）TestAug的测试用例更加多样化；（3）TestAug在创建测试套件方面大大节省了人力成本。

TestAug：一个增强能力为基础的NLP测试框架