Large Language Models (LLMs) are advancing at a rapid pace, with significant
improvements at natural language processing and coding tasks. Yet, their
ability to work with formal languages representing data, specifically within
the realm of knowledge graph engineering, remains under-investigated. To
evaluate the proficiency of various LLMs, we created a set of five tasks that
probe their ability to parse, understand, analyze, and create knowledge graphs
serialized in Turtle syntax. These tasks, each embodying distinct degrees of
complexity and being able to scale with the size of the problem, have been
integrated into our automated evaluation system, the LLM-KG-Bench. The
evaluation encompassed four commercially available LLMs - GPT-3.5, GPT-4,
Claude 1.3, and Claude 2.0, as well as two freely accessible offline models,
GPT4All Vicuna and GPT4All Falcon 13B. This analysis offers an in-depth
understanding of the strengths and shortcomings of LLMs in relation to their
application within RDF knowledge graph engineering workflows utilizing Turtle
representation. While our findings show that the latest commercial models
outperform their forerunners in terms of proficiency with the Turtle language,
they also reveal an apparent weakness. These models fall short when it comes to
adhering strictly to the output formatting constraints, a crucial requirement
in this context.

大型语言模型对 RDF 知识图谱工程中使用的 Turtle 语法的处理能力进行了深入研究，发现最新的商业模型在处理 Turtle 语法方面的熟练度方面优于早期模型，但在输出格式约束方面存在缺陷。

评估大型语言模型在 RDF 知识图谱的创建和理解能力方面的表现：LLMs 对 Turtle 的表达能力如何？

Benchmarking the Abilities of Large Language Models for RDF Knowledge  Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?

As the field of Large Language Models (LLMs) evolves at an accelerated pace,
the critical need to assess and monitor their performance emerges. We introduce
a benchmarking framework focused on knowledge graph engineering (KGE)
accompanied by three challenges addressing syntax and error correction, facts
extraction and dataset generation. We show that while being a useful tool, LLMs
are yet unfit to assist in knowledge graph generation with zero-shot prompting.
Consequently, our LLM-KG-Bench framework provides automatic evaluation and
storage of LLM responses as well as statistical data and visualization tools to
support tracking of prompt engineering and model performance.

大型语言模型在知识图谱工程方面的性能评估和监控是一个重要问题，我们引入了一个基于知识图谱工程的基准评测框架，涵盖了语法和纠错、事实抽取以及数据集生成三个挑战。通过实验证明，尽管具有一定的实用性，大型语言模型在零样本生成知识图谱方面尚不适用。因此，我们的 LLM-KG-Bench 框架提供了 LLM 响应的自动评估和存储机制，以及统计数据和可视化工具，以支持指导数据和模型性能的追踪。

开发一个可扩展的基准测试，用于评估大型语言模型在知识图谱工程中的表现

Developing a Scalable Benchmark for Assessing Large Language Models in  Knowledge Graph Engineering

Knowledge Graphs (KG) provide us with a structured, flexible, transparent,
cross-system, and collaborative way of organizing our knowledge and data across
various domains in society and industrial as well as scientific disciplines.
KGs surpass any other form of representation in terms of effectiveness.
However, Knowledge Graph Engineering (KGE) requires in-depth experiences of
graph structures, web technologies, existing models and vocabularies, rule
sets, logic, as well as best practices. It also demands a significant amount of
work. Considering the advancements in large language models (LLMs) and their
interfaces and applications in recent years, we have conducted comprehensive
experiments with ChatGPT to explore its potential in supporting KGE. In this
paper, we present a selection of these experiments and their results to
demonstrate how ChatGPT can assist us in the development and management of KGs.

本文介绍了最近使用 ChatGPT 对知识图谱工程进行实验的结果，探讨了其协助知识图谱的发展和管理的潜力。