In-context learning is a popular inference strategy where large language
models solve a task using only a few labelled demonstrations without needing
any parameter updates. Compared to work on monolingual (English) in-context
learning, multilingual in-context learning is under-explored, and we lack an
in-depth understanding of the role of demonstrations in this context. To
address this gap, we conduct a multidimensional analysis of multilingual
in-context learning, experimenting with 5 models from different model families,
9 datasets covering classification and generation tasks, and 56 typologically
diverse languages. Our results reveal that the effectiveness of demonstrations
varies significantly across models, tasks, and languages. We also find that
Llama 2-Chat, GPT-3.5, and GPT-4 are largely insensitive to the quality of
demonstrations. Instead, a carefully crafted template often eliminates the
benefits of demonstrations for some tasks and languages altogether. These
findings show that the importance of demonstrations might be overestimated. Our
work highlights the need for granular evaluation across multiple axes towards a
better understanding of in-context learning.

通过多维度分析多语言上下文学习对不同模型、任务和语言的效果，发现演示的有效性在各个方面有显著差异，并且发现 Llama 2-Chat、GPT-3.5 和 GPT-4 对演示的质量基本不敏感。相反，精心设计的模板往往完全消除了某些任务和语言的演示效果，这表明可能高估了演示的重要性。我们的工作强调了在多个方面进行细致评估以更好地理解上下文学习的需求。