While large language models (LLMs) have achieved impressive performance
across diverse tasks, recent studies showcase that causal LLMs suffer from the
"reversal curse". It is a typical example that the model knows "A's father is
B", but is unable to reason "B's child is A". This limitation poses a challenge
to the advancement of artificial general intelligence (AGI), as it suggests a
gap in the models' ability to comprehend and apply bidirectional reasoning. In
this paper, we first conduct substantial evaluation and identify that the root
cause of the reversal curse lies in the different word order between the
training and inference stage, namely, the poor ability of causal language
models to predict antecedent words within the training data. Accordingly,
permutation on the training data is considered as a potential solution, since
this can make the model predict antecedent words or tokens. However, previous
permutation methods may disrupt complete phrases or entities, thereby posing
challenges for the model to comprehend and learn from training data. To address
this issue, we propose Semantic-aware Permutation Training (SPT), which
addresses this issue by segmenting the training sentences into semantic units
(i.e., entities or phrases) with an assistant language model and permuting
these units before feeding into the model. Extensive experiments demonstrate
that SPT effectively mitigates the reversal curse since the performance on
reversed questions approximates that on the forward ones, and significantly
advances the performance of existing works.

通过语义感知排列训练（SPT）方法，我们成功解决了因句子结构差异导致的逆转诅咒问题，改进了因果语言模型的性能，并推进了当前研究的发展。

通过语义感知的排列训练减轻逆序诅咒

Mitigating Reversal Curse via Semantic-aware Permutation Training

Large language models (LLMs), such as ChatGPT and GPT4, are making new waves
in the field of natural language processing and artificial intelligence, due to
their emergent ability and generalizability. However, LLMs are black-box
models, which often fall short of capturing and accessing factual knowledge. In
contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are
structured knowledge models that explicitly store rich factual knowledge. KGs
can enhance LLMs by providing external knowledge for inference and
interpretability. Meanwhile, KGs are difficult to construct and evolving by
nature, which challenges the existing methods in KGs to generate new facts and
represent unseen knowledge. Therefore, it is complementary to unify LLMs and
KGs together and simultaneously leverage their advantages. In this article, we
present a forward-looking roadmap for the unification of LLMs and KGs. Our
roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs,
which incorporate KGs during the pre-training and inference phases of LLMs, or
for the purpose of enhancing understanding of the knowledge learned by LLMs; 2)
LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding,
completion, construction, graph-to-text generation, and question answering; and
3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a
mutually beneficial way to enhance both LLMs and KGs for bidirectional
reasoning driven by both data and knowledge. We review and summarize existing
efforts within these three frameworks in our roadmap and pinpoint their future
research directions.

本文提出了使用知识图谱和大型语言模型的三种框架，分别为增强了的知识图谱、增强了的大型语言模型和协同增强。这三个框架能够互相协作，实现双向推理，提高自然语言处理和人工智能的表现，同时指出未来的研究方向。