Current approaches to evaluating large language models (LLMs) with
pre-existing Knowledge Graphs (KG) mostly ignore the structure of the KG and
make arbitrary choices of which part of the graph to evaluate. In this paper,
we introduce KGLens, a method to evaluate LLMs by generating natural language
questions from a KG in a structure aware manner so that we can characterize its
performance on a more aggregated level. KGLens uses a parameterized KG, where
each edge is augmented with a beta distribution that guides how to sample edges
from the KG for QA testing. As the evaluation proceeds, different edges of the
parameterized KG are sampled and assessed appropriately, converging to a more
global picture of the performance of the LLMs on the KG as a whole. In our
experiments, we construct three domain-specific KGs for knowledge assessment,
comprising over 19,000 edges, 700 relations, and 21,000 entities. The results
demonstrate that KGLens can not only assess overall performance but also
provide topic, temporal, and relation analyses of LLMs. This showcases the
adaptability and customizability of KGLens, emphasizing its ability to focus
the evaluation based on specific criteria.

KGLens 是一种通过在结构感知的方式下从知识图谱生成自然语言问题以评估大型语言模型的方法，可以对模型的性能进行更合聚的整体性评估，并提供主题、时间和关系分析。

KGLens：参数化的知识图谱解决方案评估 LLM 所知和不知

KGLens: A Parameterized Knowledge Graph Solution to Assess What an LLM  Does and Doesn't Know

Accurately recommending candidate news articles to users is a basic challenge
faced by personalized news recommendation systems. Traditional methods are
usually difficult to grasp the complex semantic information in news texts,
resulting in unsatisfactory recommendation results. Besides, these traditional
methods are more friendly to active users with rich historical behaviors.
However, they can not effectively solve the "long tail problem" of inactive
users. To address these issues, this research presents a novel general
framework that combines Large Language Models (LLM) and Knowledge Graphs (KG)
into semantic representations of traditional methods. In order to improve
semantic understanding in complex news texts, we use LLMs' powerful text
understanding ability to generate news representations containing rich semantic
information. In addition, our method combines the information about news
entities and mines high-order structural information through multiple hops in
KG, thus alleviating the challenge of long tail distribution. Experimental
results demonstrate that compared with various traditional models, the
framework significantly improves the recommendation effect. The successful
integration of LLM and KG in our framework has established a feasible path for
achieving more accurate personalized recommendations in the news field. Our
code is available at this https URL

通过将大型语言模型（LLM）和知识图谱（KG）结合为传统方法的语义表示形式，该研究提出了一个新的通用框架，以准确地推荐候选新闻文章给用户，从而改善了新闻推荐系统中的个性化推荐效果。