With the advent of large language models (LLMs), the trend in NLP has been to
train LLMs on vast amounts of data to solve diverse language understanding and
generation tasks. The list of LLM successes is long and varied. Nevertheless,
several recent papers provide empirical evidence that LLMs fail to capture
important aspects of linguistic meaning. Focusing on universal quantification,
we provide a theoretical foundation for these empirical findings by proving
that LLMs cannot learn certain fundamental semantic properties including
semantic entailment and consistency as they are defined in formal semantics.
More generally, we show that LLMs are unable to learn concepts beyond the first
level of the Borel Hierarchy, which imposes severe limits on the ability of
LMs, both large and small, to capture many aspects of linguistic meaning. This
means that LLMs will continue to operate without formal guarantees on tasks
that require entailments and deep linguistic understanding.

本文研究大语言模型在自然语言处理中的局限性，特别是无法学习一些基本语义属性，如语义蕴涵和一致性，以及不能学习超越 Borel 层次结构的概念，这对语言模型的语言理解能力产生了严重限制。

使用语言模型的学习界限

Limits for Learning with Language Models

Neural networks using distributed representations can benefit from computing
semantic entailments because they enable finding contradictions, implied
knowledge, or computing plans on how to achieve distant goals. We developed
FALCON, a Fuzzy ALC Ontology Neural reasoner. FALCON uses fuzzy logic operators
to generate model structures for arbitrary ALC ontologies, and uses multiple
model structures to compute semantic entailments. Theoretical results show that
FALCON approximates semantic entailment over ALC ontologies and therefore
endows neural networks with world models and the ability to reason over them.
Experimental results show that FALCON enables approximate reasoning,
paraconsistent reasoning (reasoning with inconsistencies), and improves machine
learning in the biomedical domain by incorporating background knowledge.

使用模糊逻辑算子生成任意 ALC 本体的模型结构并使用多个模型结构计算语义蕴涵，FALCON 可以实现对 ALC 本体的近似语义蕴涵，从而赋予神经网络世界模型和推理能力，提升生物医学领域中的机器学习。

FALCON: 基于 ALC 本体的神经语义蕴含模型

FALCON: Faithful Neural Semantic Entailment over ALC Ontologies

Many recent successes in sentence representation learning have been achieved
by simply fine-tuning on the Natural Language Inference (NLI) datasets with
triplet loss or siamese loss. Nevertheless, they share a common weakness:
sentences in a contradiction pair are not necessarily from different semantic
categories. Therefore, optimizing the semantic entailment and contradiction
reasoning objective alone is inadequate to capture the high-level semantic
structure. The drawback is compounded by the fact that the vanilla siamese or
triplet losses only learn from individual sentence pairs or triplets, which
often suffer from bad local optima. In this paper, we propose PairSupCon, an
instance discrimination based approach aiming to bridge semantic entailment and
contradiction understanding with high-level categorical concept encoding. We
evaluate PairSupCon on various downstream tasks that involve understanding
sentence semantics at different granularities. We outperform the previous
state-of-the-art method with $10\%$--$13\%$ averaged improvement on eight
clustering tasks, and $5\%$--$6\%$ averaged improvement on seven semantic
textual similarity (STS) tasks.

本文提出了一种基于实例区分的方法 PairSupCon，旨在通过高级分类概念编码在语义蕴含和矛盾理解之间建立联系。该方法在各种下游任务中均取得了极好的效果，超过了先前的最先进方法。