An important assumption that comes with using LLMs on psycholinguistic data
has gone unverified. LLM-based predictions are based on subword tokenization,
not decomposition of words into morphemes. Does that matter? We carefully test
this by comparing surprisal estimates using orthographic, morphological, and
BPE tokenization against reading time data. Our results replicate previous
findings and provide evidence that in the aggregate, predictions using BPE
tokenization do not suffer relative to morphological and orthographic
segmentation. However, a finer-grained analysis points to potential issues with
relying on BPE-based tokenization, as well as providing promising results
involving morphologically-aware surprisal estimates and suggesting a new method
for evaluating morphological prediction.

基于 LLMs 的预测基于子词标记而不是词素分解，我们通过比较使用拼写、词素和 BPE 标记的惊异估计与阅读时间数据来仔细测试这一重要假设，我们的结果重现了以前的研究发现并提供了证据，表明使用 BPE 标记的预测相对于词法和拼写分割并没有出现问题，但经过更详细的分析发现依赖于 BPE 标记可能存在潜在问题，同时也提供了相关研究的有希望结果，并提出了一种评估词素预测的新方法。