Sentence and word embeddings encode structural and semantic information in a
distributed manner. Part of the information encoded -- particularly lexical
information -- can be seen as continuous, whereas other -- like structural
information -- is most often discrete. We explore whether we can compress
transformer-based sentence embeddings into a representation that separates
different linguistic signals -- in particular, information relevant to
subject-verb agreement and verb alternations. We show that by compressing an
input sequence that shares a targeted phenomenon into the latent layer of a
variational autoencoder-like system, the targeted linguistic information
becomes more explicit. A latent layer with both discrete and continuous
components captures better the targeted phenomena than a latent layer with only
discrete or only continuous components. These experiments are a step towards
separating linguistic signals from distributed text embeddings and linking them
to more symbolic representations.

我们研究了如何将基于 transformer 的句子嵌入压缩成一种表示，以分离不同的语言信号，特别是与主谓一致和谓词交替相关的信息。通过在变分自编码器类似系统的潜在层中压缩共享目标现象的输入序列，我们发现目标语言信息变得更加明确。具有离散和连续组成部分的潜在层比只有离散或只有连续组成部分的潜在层更好地捕捉目标现象。这些实验是将分布式文本嵌入中的语言信号与更符号化的表示相分离并将它们联系起来的一步。

在基于 Transformer 的句子嵌入中解析连续与离散的语言信号

Disentangling continuous and discrete linguistic signals in  transformer-based sentence embeddings

Given the increasing realism of social interactions online, social media
offers an unprecedented avenue to evaluate real-life moral scenarios. We
examine posts from Reddit, where authors and commenters share their moral
judgments on who is blameworthy. We employ computational techniques to
investigate factors influencing moral judgments, including (1) events
activating social commonsense and (2) linguistic signals. To this end, we focus
on excerpt-which we term moral sparks-from original posts that commenters
include to indicate what motivates their moral judgments. By examining over
24,672 posts and 175,988 comments, we find that event-related negative personal
traits (e.g., immature and rude) attract attention and stimulate blame,
implying a dependent relationship between moral sparks and blameworthiness.
Moreover, language that impacts commenters' cognitive processes to depict
events and characters enhances the probability of an excerpt become a moral
spark, while factual and concrete descriptions tend to inhibit this effect.

社交媒体通过对现实道德场景的评估提供了一个前所未有的途径，我们使用计算方法研究影响道德判断的因素，其中包括触发社交常识的事件和语言信号。通过对超过 24,672 个帖子和 175,988 个评论的研究，我们发现与事件相关的负面个人特质吸引了关注并激发了指责，这表明道德火花与应受指责之间存在相互依赖的关系。此外，影响评论者的认知过程来描绘事件和人物的语言有助于成为道德火花的可能性，而事实和具体的描述则倾向于抑制这种效果。

Reddit 中关于道德判断的叙事研究：通过社会常识和语言信号调查道德火花

Moral Judgments in Narratives on Reddit: Investigating Moral Sparks via  Social Commonsense and Linguistic Signals

The presence of specific linguistic signals particular to a certain sub-group
of people can be picked up by language models during training. This may lead to
discrimination if the model has learnt to pick up on a certain group's
language. If the model begins to associate specific language with a distinct
group, any decisions made based upon this language would hold a strong
correlation to a decision based on their protected characteristic. We explore a
possible technique for bias mitigation in the form of simplification of text.
The driving force of this idea is that simplifying text should standardise
language to one way of speaking while keeping the same meaning. The experiment
shows promising results as the classifier accuracy for predicting the sensitive
attribute drops by up to 17% for the simplified data.

对语言模型所训练的某些人群的特定语言信号的存在进行研究，发现这可能导致歧视。本文探讨一种减少偏见的可能技术，即简化文本，实验结果表明，对于简化的数据，预测敏感属性的分类器准确性下降了最多 17%。