Large language models (LLMs) have demonstrated remarkable performance in a
range of natural language understanding and generation tasks. Yet, their
ability to generate counterfactuals, which can be used for areas like data
augmentation, remains under-explored. This study aims to investigate the
counterfactual generation capabilities of LLMs and analysis factors that
influence this ability. First, we evaluate how effective are LLMs in
counterfactual generation through data augmentation experiments for small
language models (SLMs) across four tasks: sentiment analysis, natural language
inference, named entity recognition, and relation extraction. While LLMs show
promising enhancements in various settings, they struggle in complex tasks due
to their self-limitations and the lack of logical guidance to produce
counterfactuals that align with commonsense. Second, our analysis reveals the
pivotal role of providing accurate task definitions and detailed step-by-step
instructions to LLMs in generating counterfactuals. Interestingly, we also find
that LLMs can generate reasonable counterfactuals even with unreasonable
demonstrations, which illustrates that demonstrations are primarily to regulate
the output format.This study provides the first comprehensive insight into
counterfactual generation abilities of LLMs, and offers a novel perspective on
utilizing LLMs for data augmentation to enhance SLMs.

本研究探究了大型语言模型（LLMs）进行反事实生成和数据增强的能力，发现 LLMs 很有效的进行反事实生成，但因为自身限制和缺少实用逻辑指导，在某些复杂任务上表现欠佳。提供准确的任务定义和详细的操作步骤对于在 LLMs 中生成反事实具有决定性影响。同时，研究还证明了 LLMs 可能在缺乏合理演示的情况下生成合理的反事实，说明演示主要用于规范输出格式。因此，基于 LLMs 进行数据增强来增强 SLMs 是一种非常有前景的研究和应用方向。