Language models have become nearly ubiquitous in natural language processing
applications achieving state-of-the-art results in many tasks including
prosody. As the model design does not define predetermined linguistic targets
during training but rather aims at learning generalized representations of the
language, analyzing and interpreting the representations that models implicitly
capture is important in bridging the gap between interpretability and model
performance. Several studies have explored the linguistic information that
models capture providing some insights on their representational capacity.
However, the current studies have not explored whether prosody is part of the
structural information of the language that models learn. In this work, we
perform a series of experiments on BERT probing the representations captured at
different layers. Our results show that information about prosodic prominence
spans across many layers but is mostly focused in middle layers suggesting that
BERT relies mostly on syntactic and semantic information.

本研究采用了一系列实验，探究了 BERT 模型在不同层次上的语言表征，结果显示它主要依赖句法和语义信息来捕捉音调重音的相关信息。

BERT 对韵律学习了什么？

What does BERT learn about prosody?

With the advent of deep learning, a huge number of text-to-speech (TTS)
models which produce human-like speech have emerged. Recently, by introducing
syntactic and semantic information w.r.t the input text, various approaches
have been proposed to enrich the naturalness and expressiveness of TTS models.
Although these strategies showed impressive results, they still have some
limitations in utilizing language information. First, most approaches only use
graph networks to utilize syntactic and semantic information without
considering linguistic features. Second, most previous works do not explicitly
consider adjacent words when encoding syntactic and semantic information, even
though it is obvious that adjacent words are usually meaningful when encoding
the current word. To address these issues, we propose Relation-aware Word
Encoding Network (RWEN), which effectively allows syntactic and semantic
information based on two modules (i.e., Semantic-level Relation Encoding and
Adjacent Word Relation Encoding). Experimental results show substantial
improvements compared to previous works.

本研究提出了 Relation-aware Word Encoding Network（RWEN）模型，通过两个模块（即语义级别关系编码和相邻单词关系编码）有效利用语法和语义信息，与以前的方法相比，实验结果显示出了显著的改进。

RWEN-TTS: 自然语音合成的关系感知字编码网络

RWEN-TTS: Relation-aware Word Encoding Network for Natural Text-to-Speech Synthesis

Transformer-based QG models can generate question-answer pairs (QAPs) with
high qualities, but may also generate silly questions for certain texts. We
present a new method called tag-set sequence learning to tackle this problem,
where a tag-set sequence is a sequence of tag sets to capture the syntactic and
semantic information of the underlying sentence, and a tag set consists of one
or more language feature tags, including, for example, semantic-role-labeling,
part-of-speech, named-entity-recognition, and sentiment-indication tags. We
construct a system called TSS-Learner to learn tag-set sequences from given
declarative sentences and the corresponding interrogative sentences, and derive
answers to the latter. We train a TSS-Learner model for the English language
using a small training dataset and show that it can indeed generate adequate
QAPs for certain texts that transformer-based models do poorly. Human
evaluation on the QAPs generated by TSS-Learner over SAT practice reading tests
is encouraging.

通过使用一种叫做 tag-set sequence learning 的新方法，TSS-Learner 模型可以从给定的陈述句中学习 tag-set sequence，在特定的文本中生成得到的问题和答案对，相比于 transformer-based 模型，其质量更高且更正常。

基于标签集序列学习生成问题 - 答案对

Tag-Set-Sequence Learning for Generating Question-Answer Pairs

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and
semantic information, resulting in state-of-the-art performance on various
tasks. We propose a very fast variational information bottleneck (VIB) method
to nonlinearly compress these embeddings, keeping only the information that
helps a discriminative parser. We compress each word embedding to either a
discrete tag or a continuous vector. In the discrete version, our automatically
compressed tags form an alternative tag set: we show experimentally that our
tags capture most of the information in traditional POS tag annotations, but
our tag sequences can be parsed more accurately at the same level of tag
granularity. In the continuous version, we show experimentally that moderately
compressing the word embeddings by our method yields a more accurate parser in
8 of 9 languages, unlike simple dimensionality reduction.

提出了一种 VIB 方法，将预训练的词嵌入非线性压缩，只保留帮助判别式解析器的信息，在离散或连续向量上进行压缩，可以实现更准确的解析。