Large language models (LLMs) are transforming the ways the general public
accesses and consumes information. Their influence is particularly pronounced
in pivotal sectors like healthcare, where lay individuals are increasingly
appropriating LLMs as conversational agents for everyday queries. While LLMs
demonstrate impressive language understanding and generation proficiencies,
concerns regarding their safety remain paramount in these high-stake domains.
Moreover, the development of LLMs is disproportionately focused on English. It
remains unclear how these LLMs perform in the context of non-English languages,
a gap that is critical for ensuring equity in the real-world use of these
systems.This paper provides a framework to investigate the effectiveness of
LLMs as multi-lingual dialogue systems for healthcare queries. Our
empirically-derived framework XlingEval focuses on three fundamental criteria
for evaluating LLM responses to naturalistic human-authored health-related
questions: correctness, consistency, and verifiability. Through extensive
experiments on four major global languages, including English, Spanish,
Chinese, and Hindi, spanning three expert-annotated large health Q&A datasets,
and through an amalgamation of algorithmic and human-evaluation strategies, we
found a pronounced disparity in LLM responses across these languages,
indicating a need for enhanced cross-lingual capabilities. We further propose
XlingHealth, a cross-lingual benchmark for examining the multilingual
capabilities of LLMs in the healthcare context. Our findings underscore the
pressing need to bolster the cross-lingual capacities of these models, and to
provide an equitable information ecosystem accessible to all.

本文提供了一个框架来研究 LLMs 作为多语言对话系统在医疗查询中的有效性。通过对英语、西班牙语、中文和印地语等四种主要全球语言的广泛实验和算法与人工评估相结合的策略，我们发现 LLMs 在这些语言中的回答存在明显差异，表明需要增强跨语言能力。我们进一步提出了一种用于检测 LLMs 在医疗环境中多语言能力的跨语言基准。研究结果强调了增强这些模型的跨语言能力以及提供一个对所有人可访问的公平信息生态系统的紧迫性。