Temporal Knowledge graph completion (TKGC) is a crucial task that involves
reasoning at known timestamps to complete the missing part of facts and has
attracted more and more attention in recent years. Most existing methods focus
on learning representations based on graph neural networks while inaccurately
extracting information from timestamps and insufficiently utilizing the implied
information in relations. To address these problems, we propose a novel TKGC
model, namely Pre-trained Language Model with Prompts for TKGC (PPT). We
convert a series of sampled quadruples into pre-trained language model inputs
and convert intervals between timestamps into different prompts to make
coherent sentences with implicit semantic information. We train our model with
a masking strategy to convert TKGC task into a masked token prediction task,
which can leverage the semantic information in pre-trained language models.
Experiments on three benchmark datasets and extensive analysis demonstrate that
our model has great competitiveness compared to other models with four metrics.
Our model can effectively incorporate information from temporal knowledge
graphs into the language models.

本文提出了一种基于预训练语言模型的 TKGC 模型 (PPT)，通过将时间戳之间的间隔转换为不同的提示来提取暗示的语义信息以形成连贯的句子，并使用蒙版策略将 TKGC 任务转换为蒙版标记预测任务，有效地将时间知识图中的信息整合到语言模型中，具有较高的竞争力。

带提示的预训练语言模型用于时间知识图谱补全

Pre-trained Language Model with Prompts for Temporal Knowledge Graph  Completion

Building scalable models to learn from diverse, multimodal data remains an
open challenge. For vision-language data, the dominant approaches are based on
contrastive learning objectives that train a separate encoder for each
modality. While effective, contrastive learning approaches introduce sampling
bias depending on the data augmentations used, which can degrade performance on
downstream tasks. Moreover, these methods are limited to paired image-text
data, and cannot leverage widely-available unpaired data. In this paper, we
investigate whether a large multimodal model trained purely via masked token
prediction, without using modality-specific encoders or contrastive learning,
can learn transferable representations for downstream tasks. We propose a
simple and scalable network architecture, the Multimodal Masked Autoencoder
(M3AE), which learns a unified encoder for both vision and language data via
masked token prediction. We provide an empirical study of M3AE trained on a
large-scale image-text dataset, and find that M3AE is able to learn
generalizable representations that transfer well to downstream tasks.
Surprisingly, we find that M3AE benefits from a higher text mask ratio
(50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the
joint training of two data modalities. We also provide qualitative analysis
showing that the learned representation incorporates meaningful information
from both image and language. Lastly, we demonstrate the scalability of M3AE
with larger model size and training time, and its flexibility to train on both
paired image-text data as well as unpaired data.

本论文提出一种基于 Masked Token 预测的大型多模式模型 (M3AE)，能在不引入偏好于数据增强的对比学习目标的情况下，学习出适用于下游任务的可传递表示，并证明了高文本遮盖率下训练能够显著提升模型性能。M3AE 能够在配对和非配对的图像 - 文本数据上训练，具有可扩展性和灵活性。