Topic modeling has been a widely used tool for unsupervised text analysis.
However, comprehensive evaluations of a topic model remain challenging.
Existing evaluation methods are either less comparable across different models
(e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic
quality or document representation quality) at a time, which is insufficient to
reflect the overall model performance. In this paper, we propose WALM (Words
Agreement with Language Model), a new evaluation method for topic modeling that
comprehensively considers the semantic quality of document representations and
topics in a joint manner, leveraging the power of large language models (LLMs).
With extensive experiments involving different types of topic models, WALM is
shown to align with human judgment and can serve as a complementary evaluation
method to the existing ones, bringing a new perspective to topic modeling. Our
software package will be available at
this https URL, which can be integrated
with many widely used topic models.

我们提出了 WALM (Words Agreement with Language Model) 这一新的主题建模评估方法，综合考虑语义质量，旨在提供一种全面性评估。WALM 与人员判断一致，可作为现有评估方法的补充，为主题建模带来了新的视角。

使用大型语言模型自动评估主题模型：读懂茶叶渣

LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large  Language Models

Managing the semantic quality of the categorization in large textual
datasets, such as Wikipedia, presents significant challenges in terms of
complexity and cost. In this paper, we propose leveraging transformer models to
distill semantic information from texts in the Wikipedia dataset and its
associated categories into a latent space. We then explore different approaches
based on these encodings to assess and enhance the semantic identity of the
categories. Our graphical approach is powered by Convex Hull, while we utilize
Hierarchical Navigable Small Worlds (HNSWs) for the hierarchical approach. As a
solution to the information loss caused by the dimensionality reduction, we
modulate the following mathematical solution: an exponential decay function
driven by the Euclidean distances between the high-dimensional encodings of the
textual categories. This function represents a filter built around a contextual
category and retrieves items with a certain Reconsideration Probability (RP).
Retrieving high-RP items serves as a tool for database administrators to
improve data groupings by providing recommendations and identifying outliers
within a contextual framework.

使用转换器模型从维基百科数据集及其相关类别中提取语义信息，通过不同方法对类别的语义特征进行评估和增强，以提高数据组织的语义准确性。

通过 LLMs 潜空间增强上下文分类

Contextual Categorization Enhancement through LLMs Latent-Space

Definition Modeling, the task of generating definitions, was first proposed
as a means to evaluate the semantic quality of word embeddings-a coherent
lexical semantic representations of a word in context should contain all the
information necessary to generate its definition. The relative novelty of this
task entails that we do not know which factors are actually relied upon by a
Definition Modeling system. In this paper, we present evidence that the task
may not involve as much semantics as one might expect: we show how an earlier
model from the literature is both rather insensitive to semantic aspects such
as explicit polysemy, as well as reliant on formal similarities between
headwords and words occurring in its glosses, casting doubt on the validity of
the task as a means to evaluate embeddings.

本文研究任务是生成定义，目的是评估单词嵌入的语义质量，本文证明这个任务与语义相关性较小，并且依赖于头词和语境中出现的单词之间的形式相似性，从而对用此任务来评估单词嵌入的有效性提出质疑。

定义建模：建立定义。使用少量或没有语义生成定义

"Definition Modeling: To model definitions." Generating Definitions With  Little to No Semantics

Morphologically rich languages accentuate two properties of distributional
vector space models: 1) the difficulty of inducing accurate representations for
low-frequency word forms; and 2) insensitivity to distinct lexical relations
that have similar distributional signatures. These effects are detrimental for
language understanding systems, which may infer that 'inexpensive' is a
rephrasing for 'expensive' or may not associate 'acquire' with 'acquires'. In
this work, we propose a novel morph-fitting procedure which moves past the use
of curated semantic lexicons for improving distributional vector spaces.
Instead, our method injects morphological constraints generated using simple
language-specific rules, pulling inflectional forms of the same word close
together and pushing derivational antonyms far apart. In intrinsic evaluation
over four languages, we show that our approach: 1) improves low-frequency word
estimates; and 2) boosts the semantic quality of the entire word vector
collection. Finally, we show that morph-fitted vectors yield large gains in the
downstream task of dialogue state tracking, highlighting the importance of
morphology for tackling long-tail phenomena in language understanding tasks.

提出了一种新颖的基于词形的约束方法，通过使用简单的语言特定规则来改进分布向量空间的语义质量，这种方法可以提高低频词估计的准确性，并在解决语言理解任务的长尾现象方面取得显著的效果。