In this work we build upon negative results from an attempt at language
modeling with predicted semantic structure, in order to establish empirical
lower bounds on what could have made the attempt successful. More specifically,
we design a concise binary vector representation of semantic structure at the
lexical level and evaluate in-depth how good an incremental tagger needs to be
in order to achieve better-than-baseline performance with an end-to-end
semantic-bootstrapping language model. We envision such a system as consisting
of a (pretrained) sequential-neural component and a hierarchical-symbolic
component working together to generate text with low surprisal and high
linguistic interpretability. We find that (a) dimensionality of the semantic
vector representation can be dramatically reduced without losing its main
advantages and (b) lower bounds on prediction quality cannot be established via
a single score alone, but need to take the distributions of signal and noise
into account.

通过在词汇层面上构建语义结构的二元向量表示方法，研究并总结了一个基于神经组件和层次符号组件相互配合的语义引导语言模型在提高文本生成的可预测性和语言解释性上的表现所必须具备的条件，包括语义向量表示的维度，以及对这些表示结果涉及噪声和信号的分布等。

基于本地引导语义结构的语言建模的经验充分下限

Empirical Sufficiency Lower Bounds for Language Modeling with  Locally-Bootstrapped Semantic Structures

Recent advances in large language models have prompted researchers to examine
their abilities across a variety of linguistic tasks, but little has been done
to investigate how models handle the interactions in meaning across words and
larger syntactic forms -- i.e. phenomena at the intersection of syntax and
semantics. We present the semantic notion of agentivity as a case study for
probing such interactions. We created a novel evaluation dataset by utilitizing
the unique linguistic properties of a subset of optionally transitive English
verbs. This dataset was used to prompt varying sizes of three model classes to
see if they are sensitive to agentivity at the lexical level, and if they can
appropriately employ these word-level priors given a specific syntactic
context. Overall, GPT-3 text-davinci-003 performs extremely well across all
experiments, outperforming all other models tested by far. In fact, the results
are even better correlated with human judgements than both syntactic and
semantic corpus statistics. This suggests that LMs may potentially serve as
more useful tools for linguistic annotation, theory testing, and discovery than
select corpora for certain tasks.

以 semantic notion of agentivity 为例，研究了 large language models 在句法和语义交互方面的表现；作者创建了一个新颖的评估数据集，发现 GPT-3 的表现非常好，甚至比语法和语义语料库的统计结果还要好，这表明大语言模型可能对语言注释、理论测试和发现比某些任务的语料库更有用。

句法与语义的交汇：通过代理性探究语言模型的句法 - 语义接口

Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics  Interface of LMs Through Agentivity

Two fundamental questions in neurolinguistics concerns the brain regions that
integrate information beyond the lexical level, and the size of their window of
integration. To address these questions we introduce a new approach named
masked-attention generation. It uses GPT-2 transformers to generate word
embeddings that capture a fixed amount of contextual information. We then
tested whether these embeddings could predict fMRI brain activity in humans
listening to naturalistic text. The results showed that most of the cortex
within the language network is sensitive to contextual information, and that
the right hemisphere is more sensitive to longer contexts than the left.
Masked-attention generation supports previous analyses of context-sensitivity
in the brain, and complements them by quantifying the window size of context
integration per voxel.

利用 GPT-2 变形金刚生成的单词嵌入来测试大脑对自然文本中背景信息的反应，结果表明语言网络中的大多数皮层对背景信息敏感，右半球对于较长场景的反应比左半球更敏感，支持以前的上下文敏感度分析并有助于量化每个体素的上下文整合窗口大小。