Large language models (LLMs) have shown impressive capabilities in tasks such
as machine translation, text summarization, question answering, and solving
complex mathematical problems. However, their primary training on data-rich
languages like English limits their performance in low-resource languages. This
study addresses this gap by focusing on the Indexical Shift problem in Turkish.
The Indexical Shift problem involves resolving pronouns in indexical shift
contexts, a grammatical challenge not present in high-resource languages like
English. We present the first study examining indexical shift in any language,
releasing a Turkish dataset specifically designed for this purpose. Our
Indexical Shift Dataset consists of 156 multiple-choice questions, each
annotated with necessary linguistic details, to evaluate LLMs in a few-shot
setting. We evaluate recent multilingual LLMs, including GPT-4, GPT-3.5,
Cohere-AYA, Trendyol-LLM, and Turkcell-LLM, using this dataset. Our analysis
reveals that even advanced models like GPT-4 struggle with the grammatical
nuances of indexical shift in Turkish, achieving only moderate performance.
These findings underscore the need for focused research on the grammatical
challenges posed by low-resource languages. We released the dataset and code
\href{this https URL} {here}.

通过研究指代转换问题，本研究以土耳其语为例，评估了多语种大型语言模型在低资源语言中解决语法挑战的能力，并发布了专门设计用于此目的的土耳其数据集，发现即使是先进的模型如 GPT-4 在土耳其语的指代转换问题上也仅能达到中等性能。