Large language models (LLMs) often lack culture-specific knowledge of daily
life, especially across diverse regions and non-English languages. Existing
benchmarks for evaluating LLMs' cultural sensitivities are limited to a single
language or collected from online sources such as Wikipedia, which do not
reflect the mundane everyday lifestyles of diverse regions. That is,
information about the food people eat for their birthday celebrations, spices
they typically use, musical instruments youngsters play, or the sports they
practice in school is common cultural knowledge but uncommon in easily
collected online sources, especially for underrepresented cultures. To address
this issue, we introduce BLEnD, a hand-crafted benchmark designed to evaluate
LLMs' everyday knowledge across diverse cultures and languages. BLEnD comprises
52.6k question-answer pairs from 16 countries/regions, in 13 different
languages, including low-resource ones such as Amharic, Assamese, Azerbaijani,
Hausa, and Sundanese. We construct the benchmark to include two formats of
questions: short-answer and multiple-choice. We show that LLMs perform better
for cultures that are highly represented online, with a maximum 57.34%
difference in GPT-4, the best-performing model, in the short-answer format. For
cultures represented by mid-to-high-resource languages, LLMs perform better in
their local languages, but for cultures represented by low-resource languages,
LLMs perform better in English than the local languages. We make our dataset
publicly available at: this https URL

LLMs 在文化敏感性方面缺乏对日常生活的文化特定知识，不同于在线来源如维基百科，BLEnD 通过手工设计的基准评估了 LLMs 对多元文化和语言的日常知识，结果显示 LLMs 在高度在线代表的文化中表现更好，而在低资源语言文化中表现更好的是英语而非本地语言。

BLEnD: 基于多元文化和语言的常识性 LLM 基准

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures  and Languages

We present an approach to generate novel computer game levels that blend
different game concepts in an unsupervised fashion. Our primary contribution is
an analogical reasoning process to construct blends between level design models
learned from gameplay videos. The models represent probabilistic relationships
between elements in the game. An analogical reasoning process maps features
between two models to produce blended models that can then generate new level
chunks. As a proof-of-concept we train our system on the classic platformer
game Super Mario Bros. due to its highly-regarded and well understood level
design. We evaluate the extent to which the models represent stylistic level
design knowledge and demonstrate the ability of our system to explain levels
that were blended by human expert designers.

本文提出了一种无监督融合不同游戏概念生成新的计算机游戏关卡的方法，主要贡献在于引入类比推理过程，以构建从游戏视频中学习的不同层级设计模型之间的融合模型，并通过证明在超级马里奥兄弟游戏中表现出的风格化水平设计知识的方式，演示了我们的系统说明了由人类专家设计者融合的游戏关卡。