Text documents, including programs, typically have human-readable semantic
structure. Historically, programmatic access to these semantics has required
explicit in-document tagging. Especially in systems where the text has an
execution semantics, this means it is an opt-in feature that is hard to support
properly. Today, language models offer a new method: metadata can be bound to
entities in changing text using a model's human-like understanding of
semantics, with no requirements on the document structure. This method expands
the applications of document annotation, a fundamental operation in program
writing, debugging, maintenance, and presentation. We contribute a system that
employs an intelligent agent to re-tag modified programs, enabling rich
annotations to automatically follow code as it evolves. We also contribute a
formal problem definition, an empirical synthetic benchmark suite, and our
benchmark generator. Our system achieves an accuracy of 90% on our benchmarks
and can replace a document's tags in parallel at a rate of 5 seconds per tag.
While there remains significant room for improvement, we find performance
reliable enough to justify further exploration of applications.

这篇论文研究了使用语言模型在文档中绑定元数据的方法，以实现自动标记和丰富注释的功能，并通过实验验证了系统的准确性和性能。

魔幻标记：利用 LLM 维护文档外部标记

Magic Markup: Maintaining Document-External Markup with an LLM

Topic models are a popular tool for understanding text collections, but their
evaluation has been a point of contention. Automated evaluation metrics such as
coherence are often used, however, their validity has been questioned for
neural topic models (NTMs) and can overlook the benefits of a model in real
world applications. To this end, we conduct the first evaluation of neural,
supervised and classical topic models in an interactive task based setting. We
combine topic models with a classifier and test their ability to help humans
conduct content analysis and document annotation. From simulated, real user and
expert pilot studies, the Contextual Neural Topic Model does the best on
cluster evaluation metrics and human evaluations; however, LDA is competitive
with two other NTMs under our simulated experiment and user study results,
contrary to what coherence scores suggest. We show that current automated
metrics do not provide a complete picture of topic modeling capabilities, but
the right choice of NTMs can be better than classical models on practical
tasks.

通过对话题模型进行评估，发现神经网络模型（NTMs）在实际任务中的效果优于传统模型，且自动评估指标无法提供全面的话题建模能力。