Understanding which information is encoded in deep models of spoken and
written language has been the focus of much research in recent years, as it is
crucial for debugging and improving these architectures. Most previous work has
focused on probing for speaker characteristics, acoustic and phonological
information in models of spoken language, and for syntactic information in
models of written language. Here we focus on the encoding of syntax in several
self-supervised and visually grounded models of spoken language. We employ two
complementary probing methods, combined with baselines and reference
representations to quantify the degree to which syntactic structure is encoded
in the activations of the target models. We show that syntax is captured most
prominently in the middle layers of the networks, and more explicitly within
models with more parameters.

研究了自我监督和视觉基础的多个口语模型对句法的编码方式，证明句法结构被最明显地捕获在中间层，并在参数较多的模型中表现得更加明显。

从声音到语法：探究口语语言模型的句法

Wave to Syntax: Probing spoken language models for syntax

As speakers turn their thoughts into sentences, they maintain a balance
between the complexity of words and syntax. However, it is unclear whether this
syntax-lexicon tradeoff is unique to the spoken language production that is
under the pressure of rapid online processing. Alternatively, it is possible
that the tradeoff is a basic property of language irrespective of the modality
of production. This work evaluates the relationship between the complexity of
words and syntactic rules in the written language of neurotypical individuals
on three different topics. We found that similar to speaking, constructing
sentences in writing involves a tradeoff between the complexity of the lexical
and syntactic items. We also show that the reduced online processing demands
during writing allows for retrieving more complex words at the cost of
incorporating simpler syntax. This work further highlights the role of
accessibility of the elements of a sentence as the driving force in the
emergence of the syntax-lexicon tradeoff.

研究了典型人类书写中的语言产出、语法与词汇复杂度之间的平衡，发现与语言口语类似，书写句子也存在语法与词汇复杂度之间的回报关系，并且与语音产出不同的是，书写过程可以检索到更复杂的词汇，但会牺牲语法结构的复杂度。该研究强调了句子成分易于接近是导致语法与词汇数量之间关系出现的驱动因素。

写作中的语法与词汇权衡

The syntax-lexicon tradeoff in writing

Written language is a complex communication signal capable of conveying
information encoded in the form of ordered sequences of words. Beyond the local
order ruled by grammar, semantic and thematic structures affect long-range
patterns in word usage. Here, we show that a direct application of information
theory quantifies the relationship between the statistical distribution of
words and the semantic content of the text. We show that there is a
characteristic scale, roughly around a few thousand words, which establishes
the typical size of the most informative segments in written language.
Moreover, we find that the words whose contributions to the overall information
is larger, are the ones more closely associated with the main subjects and
topics of the text. This scenario can be explained by a model of word usage
that assumes that words are distributed along the text in domains of a
characteristic size where their frequency is higher than elsewhere. Our
conclusions are based on the analysis of a large database of written language,
diverse in subjects and styles, and thus are likely to be applicable to general
language sequences encoding complex information.

本文研究书面语的信息编码和语义特征，利用信息论的方法分析书面语中词汇的分布和使用，探求了主题 - 词汇频率关系。研究结果表明，在每个文本中，频率高的词汇与主题的相关性更大，这种现象可以用特征大小和分布模型来解释。