We tested the robustness of sarcasm detection models by examining their
behavior when fine-tuned on four sarcasm datasets containing varying
characteristics of sarcasm: label source (authors vs. third-party), domain
(social media/online vs. offline conversations/dialogues), style (aggressive
vs. humorous mocking). We tested their prediction performance on the same
dataset (intra-dataset) and across different datasets (cross-dataset). For
intra-dataset predictions, models consistently performed better when fine-tuned
with third-party labels rather than with author labels. For cross-dataset
predictions, most models failed to generalize well to the other datasets,
implying that one type of dataset cannot represent all sorts of sarcasm with
different styles and domains. Compared to the existing datasets, models
fine-tuned on the new dataset we release in this work showed the highest
generalizability to other datasets. With a manual inspection of the datasets
and post-hoc analysis, we attributed the difficulty in generalization to the
fact that sarcasm actually comes in different domains and styles. We argue that
future sarcasm research should take the broad scope of sarcasm into account.

对四个具有不同讽刺特征的讽刺数据集进行精调的讽刺检测模型的鲁棒性进行了测试，结果发现使用第三方标签进行精调的模型在内部数据集预测中表现更好，而大多数模型在跨数据集预测中无法很好地进行泛化，暗示一个数据集类型无法代表不同风格和领域的所有种类的讽刺。与现有数据集相比，本研究发布的新数据集上精调的模型在与其他数据集的泛化性能上表现最好。通过手动检查数据集和事后分析，我们发现泛化困难的原因在于讽刺实际上存在不同的领域和风格。我们认为未来的讽刺研究应考虑讽刺的广泛范围。