Lexical ambiguity -- where a single wordform takes on distinct,
context-dependent meanings -- serves as a useful tool to compare across
different large language models' (LLMs') ability to form distinct,
contextualized representations of the same stimulus. Few studies have
systematically compared LLMs' contextualized word embeddings for languages
beyond English. Here, we evaluate multiple bidirectional transformers' (BERTs')
semantic representations of Spanish ambiguous nouns in context. We develop a
novel dataset of minimal-pair sentences evoking the same or different sense for
a target ambiguous noun. In a pre-registered study, we collect contextualized
human relatedness judgments for each sentence pair. We find that various
BERT-based LLMs' contextualized semantic representations capture some variance
in human judgments but fall short of the human benchmark, and for Spanish --
unlike English -- model scale is uncorrelated with performance. We also
identify stereotyped trajectories of target noun disambiguation as a proportion
of traversal through a given LLM family's architecture, which we partially
replicate in English. We contribute (1) a dataset of controlled, Spanish
sentence stimuli with human relatedness norms, and (2) to our evolving
understanding of the impact that LLM specification (architectures, training
protocols) exerts on contextualized embeddings.

通过比较多种 BERT-based 语言模型中的语境化词嵌入，我们评估了西班牙语歧义名词的语义表达。我们开发了一个新颖的句子数据集，并收集了人类的相关性判断。结果显示，这些语言模型的语义表达在人类判断中捕捉到一些差异，但不能达到人类水平。与英语不同，我们发现在西班牙语中，模型规模与性能之间没有相关性。此外，我们还发现了目标名词消歧的陈规轨迹，并在英语中部分复制了这一结果。我们贡献了（1）一组包含人类相关性判断的西班牙语句子刺激数据集，以及（2）认识到语言模型规格（结构，训练方案）对语境化嵌入的影响。