Large Language Models (LLMs) like ChatGPT or Bard have revolutionized
information retrieval and captivated the audience with their ability to
generate custom responses in record time, regardless of the topic. In this
article, we assess the capabilities of various LLMs in producing reliable,
comprehensive, and sufficiently relevant responses about historical facts in
French. To achieve this, we constructed a testbed comprising numerous
history-related questions of varying types, themes, and levels of difficulty.
Our evaluation of responses from ten selected LLMs reveals numerous
shortcomings in both substance and form. Beyond an overall insufficient
accuracy rate, we highlight uneven treatment of the French language, as well as
issues related to verbosity and inconsistency in the responses provided by
LLMs.

通过对十个选择的大型语言模型在法语历史事实方面的回应进行评估，我们发现大型语言模型在内容和形式方面存在许多不足之处，包括整体准确率不够高、对法语的处理不均衡以及回应中的冗长和不一致性问题。

历史问题中大型语言模型（LLM）的响应能力评估

Évaluation des capacités de réponse de larges modèles de langage  (LLM) pour des questions d'historiens

Temporal knowledge graph (TKG) forecasting benchmarks challenge models to
predict future facts using knowledge of past facts. In this paper, we apply
large language models (LLMs) to these benchmarks using in-context learning
(ICL). We investigate whether and to what extent LLMs can be used for TKG
forecasting, especially without any fine-tuning or explicit modules for
capturing structural and temporal information. For our experiments, we present
a framework that converts relevant historical facts into prompts and generates
ranked predictions using token probabilities. Surprisingly, we observe that
LLMs, out-of-the-box, perform on par with state-of-the-art TKG models carefully
designed and trained for TKG forecasting. Our extensive evaluation presents
performances across several models and datasets with different characteristics,
compares alternative heuristics for preparing contextual information, and
contrasts to prominent TKG methods and simple frequency and recency baselines.
We also discover that using numerical indices instead of entity/relation names,
i.e., hiding semantic information, does not significantly affect the
performance ($\pm$0.4\% Hit@1). This shows that prior semantic knowledge is
unnecessary; instead, LLMs can leverage the existing patterns in the context to
achieve such performance. Our analysis also reveals that ICL enables LLMs to
learn irregular patterns from the historical context, going beyond simple
predictions based on common or recent information.

本文研究利用大型语言模型和上下文学习技术进行时间知识图谱（TKG）预测，并证明可在不需要显式重点捕获结构和时态信息的情况下，LLM 与针对 TKG 预测深度设计和训练的最先进的 TKG 模型表现相当，而且本文发现使用数值索引而非实体 / 关系名称可以实现接近相同的性能，且证明上下文学习可以帮助 LLMs 学习历史上不规律的模式，从而超越基于共同或最新信息的简单预测。