Curriculum learning provides a systematic approach to training. It refines
training progressively, tailors training to task requirements, and improves
generalization through exposure to diverse examples. We present a curriculum
learning approach that builds on existing knowledge about text and graph
complexity formalisms for training with text graph data. The core part of our
approach is a novel data scheduler, which employs "spaced repetition" and
complexity formalisms to guide the training process. We demonstrate the
effectiveness of the proposed approach on several text graph tasks and graph
neural network architectures. The proposed model gains more and uses less data;
consistently prefers text over graph complexity indices throughout training,
while the best curricula derived from text and graph complexity indices are
equally effective; and it learns transferable curricula across GNN models and
datasets. In addition, we find that both node-level (local) and graph-level
(global) graph complexity indices, as well as shallow and traditional text
complexity indices play a crucial role in effective curriculum learning.

基于现有的文本和图形复杂性形式主义的知识，我们提出了一种基于课程学习的方法，它通过多样性示例的暴露来逐步训练、根据任务要求精炼训练，并通过迭代训练过程中使用 “间隔重复” 和复杂性形式主义来指导训练。我们在几个文本图形任务和图形神经网络架构上展示了所提出方法的有效性。

文本图的复杂性导向课程学习

Complexity-Guided Curriculum Learning for Text Graphs

The attention mechanisms of transformers effectively extract pertinent
information from the input sequence. However, the quadratic complexity of
self-attention w.r.t the sequence length incurs heavy computational and memory
burdens, especially for tasks with long sequences. Existing accelerators face
performance degradation in these tasks. To this end, we propose SALO to enable
hybrid sparse attention mechanisms for long sequences. SALO contains a data
scheduler to map hybrid sparse attention patterns onto hardware and a spatial
accelerator to perform the efficient attention computation. We show that SALO
achieves 17.66x and 89.33x speedup on average compared to GPU and CPU
implementations, respectively, on typical workloads, i.e., Longformer and ViL.

本文提出了一种名为 SALO 的加速器，能够使得 Transformer 等注意力机制在处理长序列时具有更好的性能，并通过实验证明 SALO 相比于典型工作负载下的 GPU 和 CPU 实现，分别具有 17.66 倍和 89.33 倍的加速比。