Existing humor datasets and evaluations predominantly focus on English,
lacking resources for culturally nuanced humor in non-English languages like
Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo
Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing
intellectually challenging and culturally specific jokes. We annotate
explanations for each joke and evaluate human explanations against two
state-of-the-art LLMs, GPT-4o and ERNIE Bot, through A/B testing by native
Chinese speakers. Our evaluation shows that Chumor is challenging even for SOTA
LLMs, and the human explanations for Chumor jokes are significantly better than
explanations generated by the LLMs.

通过对 Ruo Zhi Ba 平台上的智力挑战和文化特定笑话进行注释，我们构建了一个名为 Chumor 的数据集，并通过 A/B 测试与两种先进的语言模型 GPT-4o 和 ERNIE Bot 进行人类解释评估，结果表明 Chumor 是对于 SOTA 语言模型也具有挑战性，而 Chumor 笑话的人类解释明显优于语言模型生成的解释。