We present a novel dataset for the controlled composition of counterarguments
designed for further applications in argument refining, mining, and evaluation.
Our dataset constitutes enriched counter-arguments to posts in the Reddit
ChangeMyView dataset that are integrated with evidence retrieved from
high-quality sources and generated based on user preferences, adjusting the
critical attributes of evidence and argument style. The resultant Counterfire
corpus comprises arguments generated from GPT-3.5 turbo, Koala, and PaLM 2
models and two of their finetuned variants (N = 32,000). Model evaluation
indicates strong paraphrasing abilities with evidence, albeit limited word
overlap, while demonstrating high style integration (0.9682 for 'reciprocity'),
showing the ability of LLM to assimilate diverse styles. Of all models, GPT-3.5
turbo showed the highest scores in argument quality evaluation, showing
consistent accuracy (score >0.8). In further analyses, reciprocity-style
counterarguments display higher counts in most categories, possibly indicating
a more creatively persuasive use of evidence. In contrast, human-written
counterarguments exhibited greater argumentative richness and diversity across
categories. Despite human-written arguments being favored as the most
persuasive in human evaluation, the 'No Style' generated text surprisingly
exhibited the highest score, prompting further exploration and investigation on
the trade-offs in generation for facts and style.

我们提出了一个新颖的数据集，用于控制组成反驳论点，旨在进一步应用于论证的改进、挖掘和评估。