Though there is a strong consensus that word length and frequency are the
most important single-word features determining visual-orthographic access to
the mental lexicon, there is less agreement as how to best capture syntactic
and semantic factors. The traditional approach in cognitive reading research
assumes that word predictability from sentence context is best captured by
cloze completion probability (CCP) derived from human performance data. We
review recent research suggesting that probabilistic language models provide
deeper explanations for syntactic and semantic effects than CCP. Then we
compare CCP with (1) Symbolic n-gram models consolidate syntactic and semantic
short-range relations by computing the probability of a word to occur, given
two preceding words. (2) Topic models rely on subsymbolic representations to
capture long-range semantic similarity by word co-occurrence counts in
documents. (3) In recurrent neural networks (RNNs), the subsymbolic units are
trained to predict the next word, given all preceding words in the sentences.
To examine lexical retrieval, these models were used to predict single fixation
durations and gaze durations to capture rapidly successful and standard lexical
access, and total viewing time to capture late semantic integration. The linear
item-level analyses showed greater correlations of all language models with all
eye-movement measures than CCP. Then we examined non-linear relations between
the different types of predictability and the reading times using generalized
additive models. N-gram and RNN probabilities of the present word more
consistently predicted reading performance compared with topic models or CCP.

本文研究了语言模型在理解阅读中的应用，发现概率语言模型在得知前文后更有深入的语法和语义解释，从多个层面上与阅读时间都有非常明显的关联。

语言模型比经验可预测性更好地解释单词阅读时间

Language Models Explain Word Reading Times Better Than Empirical Predictability

This work has been motivated by two long term goals: to understand how humans
learn language and to build programs that can understand language. Using a
representation that makes the relevant features explicit is a prerequisite for
successful learning and understanding. Therefore, I chose to represent
relations between individual words explicitly in my model. Lexical attraction
is defined as the likelihood of such relations. I introduce a new class of
probabilistic language models named lexical attraction models which can
represent long distance relations between words and I formalize this new class
of models using information theory.
Within the framework of lexical attraction, I developed an unsupervised
language acquisition program that learns to identify linguistic relations in a
given sentence. The only explicitly represented linguistic knowledge in the
program is lexical attraction. There is no initial grammar or lexicon built in
and the only input is raw text. Learning and processing are interdigitated. The
processor uses the regularities detected by the learner to impose structure on
the input. This structure enables the learner to detect higher level
regularities. Using this bootstrapping procedure, the program was trained on
100 million words of Associated Press material and was able to achieve 60%
precision and 50% recall in finding relations between content-words. Using
knowledge of lexical attraction, the program can identify the correct relations
in syntactically ambiguous sentences such as ``I saw the Statue of Liberty
flying over New York.''

本文介绍了一种新的基于词汇吸引力模型的自动学习自然语言的方式，无需初始化语法和词汇表，只需输入原始文本即可识别词汇间的语言关系，能够准确识别句子中的句法模糊性，并在大规模语料库中达到 60% 的准确率和 50% 的召回率。