Language Models (LMs) have proven their ability to acquire diverse linguistic
knowledge during the pretraining phase, potentially serving as a valuable
source of incidental supervision for downstream tasks. However, there has been
limited research conducted on the retrieval of domain-specific knowledge, and
specifically legal knowledge. We propose to explore the task of Entity Typing,
serving as a proxy for evaluating legal knowledge as an essential aspect of
text comprehension, and a foundational task to numerous downstream legal NLP
applications. Through systematic evaluation and analysis and two types of
prompting (cloze sentences and QA-based templates) and to clarify the nature of
these acquired cues, we compare diverse types and lengths of entities both
general and domain-specific entities, semantics or syntax signals, and
different LM pretraining corpus (generic and legal-oriented) and architectures
(encoder BERT-based and decoder-only with Llama2). We show that (1) Llama2
performs well on certain entities and exhibits potential for substantial
improvement with optimized prompt templates, (2) law-oriented LMs show
inconsistent performance, possibly due to variations in their training corpus,
(3) LMs demonstrate the ability to type entities even in the case of
multi-token entities, (4) all models struggle with entities belonging to
sub-domains of the law (5) Llama2 appears to frequently overlook syntactic
cues, a shortcoming less present in BERT-based architectures.

研究了语言模型在领域特定知识的获取和法律知识评估任务，比较了不同类型、长度和语义或语法信号的实体，不同语言模型预训练语料库和架构对于实体类型的性能影响，并发现 Llama2 在优化提示模板后表现良好，法律导向的语言模型显示性能不一致，存在的问题包括多令牌实体以及法律子领域实体的识别，此外 Llama2 在识别句法信号上存在缺陷，相较于基于 BERT 的架构缺陷更为显著。