Knowledge base construction entails acquiring structured information to
create a knowledge base of factual and relational data, facilitating question
answering, information retrieval, and semantic understanding. The challenge
called "Knowledge Base Construction from Pretrained Language Models" at
International Semantic Web Conference 2023 defines tasks focused on
constructing knowledge base using language model. Our focus was on Track 1 of
the challenge, where the parameters are constrained to a maximum of 1 billion,
and the inclusion of entity descriptions within the prompt is prohibited.
Although the masked language model offers sufficient flexibility to extend
its vocabulary, it is not inherently designed for multi-token prediction. To
address this, we present Vocabulary Expandable BERT for knowledge base
construction, which expand the language model's vocabulary while preserving
semantic embeddings for newly added words. We adopt task-specific
re-pre-training on masked language model to further enhance the language model.
Through experimentation, the results show the effectiveness of our
approaches. Our framework achieves F1 score of 0.323 on the hidden test set and
0.362 on the validation set, both data set is provided by the challenge.
Notably, our framework adopts a lightweight language model (BERT-base, 0.13
billion parameters) and surpasses the model using prompts directly on large
language model (Chatgpt-3, 175 billion parameters). Besides, Token-Recode
achieves comparable performances as Re-pretrain. This research advances
language understanding models by enabling the direct embedding of multi-token
entities, signifying a substantial step forward in link prediction task in
knowledge graph and metadata completion in data management.

知识库搭建从预训练语言模型开始，提出了一种扩展语言模型词汇量的方法，实现了多词汇实体的直接嵌入，这在知识图谱的链接预测任务和数据管理中的元数据完成方面迈出了重要的一步。