Self-supervised learning with masked autoencoders has recently gained
popularity for its ability to produce effective image or textual
representations, which can be applied to various downstream tasks without
retraining. However, we observe that the current masked autoencoder models lack
good generalization ability on graph data. To tackle this issue, we propose a
novel graph masked autoencoder framework called GiGaMAE. Different from
existing masked autoencoders that learn node presentations by explicitly
reconstructing the original graph components (e.g., features or edges), in this
paper, we propose to collaboratively reconstruct informative and integrated
latent embeddings. By considering embeddings encompassing graph topology and
attribute information as reconstruction targets, our model could capture more
generalized and comprehensive knowledge. Furthermore, we introduce a mutual
information based reconstruction loss that enables the effective reconstruction
of multiple targets. This learning objective allows us to differentiate between
the exclusive knowledge learned from a single target and common knowledge
shared by multiple targets. We evaluate our method on three downstream tasks
with seven datasets as benchmarks. Extensive experiments demonstrate the
superiority of GiGaMAE against state-of-the-art baselines. We hope our results
will shed light on the design of foundation models on graph-structured data.
Our code is available at: this https URL

我们提出了一种名为 GiGaMAE 的新型图掩蔽自编码器框架，通过协同重构信息化和综合的潜在嵌入，从而解决了当前图数据上遇到的推理能力不强的问题。通过将图拓扑和属性信息包含在嵌入中，作为重构目标，我们的模型能够捕捉到更广义和综合的知识，并引入基于互信息的重构损失，使得能够有效重构多个目标，实验结果表明 GiGaMAE 在多个基准测试中表现优于现有基线模型。

GiGaMAE: 通过协同潜在空间重建实现可泛化的图掩码自编码器

GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent  Space Reconstruction

Class incremental learning (CIL) aims to incrementally update a trained model
with the new classes of samples (plasticity) while retaining previously learned
ability (stability). To address the most challenging issue in this goal, i.e.,
catastrophic forgetting, the mainstream paradigm is memory-replay CIL, which
consolidates old knowledge by replaying a small number of old classes of
samples saved in the memory. Despite effectiveness, the inherent
destruction-reconstruction dynamics in memory-replay CIL are an intrinsic
limitation: if the old knowledge is severely destructed, it will be quite hard
to reconstruct the lossless counterpart. Our theoretical analysis shows that
the destruction of old knowledge can be effectively alleviated by balancing the
contribution of samples from the current phase and those saved in the memory.
Motivated by this theoretical finding, we propose a novel Balanced
Destruction-Reconstruction module (BDR) for memory-replay CIL, which can
achieve better knowledge reconstruction by reducing the degree of maximal
destruction of old knowledge. Specifically, to achieve a better balance between
old knowledge and new classes, the proposed BDR module takes into account two
factors: the variance in training status across different classes and the
quantity imbalance of samples from the current phase and memory. By dynamically
manipulating the gradient during training based on these factors, BDR can
effectively alleviate knowledge destruction and improve knowledge
reconstruction. Extensive experiments on a range of CIL benchmarks have shown
that as a lightweight plug-and-play module, BDR can significantly improve the
performance of existing state-of-the-art methods with good generalization.

通过平衡当前阶段的样本和内存中保存的样本的贡献，提出了一种新的平衡破坏与重建模块（BDR），用于记忆回放的类增量学习（CIL），通过减少旧知识的最大破坏程度，可以实现更好的知识重建。实验证明，作为一种轻量级即插即用模块，BDR 能显著提高现有最先进方法的性能，并具有良好的泛化能力。