Large language models record impressive performance on many natural language
processing tasks. However, their knowledge capacity is limited to the
pretraining corpus. Retrieval augmentation offers an effective solution by
retrieving context from external knowledge sources to complement the language
model. However, existing retrieval augmentation techniques ignore the
structural relationships between these documents. Furthermore, retrieval models
are not explored much in scientific tasks, especially in regard to the
faithfulness of retrieved documents. In this paper, we propose a novel
structure-aware retrieval augmented language model that accommodates document
structure during retrieval augmentation. We create a heterogeneous document
graph capturing multiple types of relationships (e.g., citation, co-authorship,
etc.) that connect documents from more than 15 scientific disciplines (e.g.,
Physics, Medicine, Chemistry, etc.). We train a graph neural network on the
curated document graph to act as a structural encoder for the corresponding
passages retrieved during the model pretraining. Particularly, along with text
embeddings of the retrieved passages, we obtain structural embeddings of the
documents (passages) and fuse them together before feeding them to the language
model. We evaluate our model extensively on various scientific benchmarks that
include science question-answering and scientific document classification
tasks. Experimental results demonstrate that structure-aware retrieval improves
retrieving more coherent, faithful and contextually relevant passages, while
showing a comparable performance in the overall accuracy.

我们提出了一种结构感知的检索增强语言模型，通过在预训练模型中检索并融合具有结构关系的科学文档，来改善检索到的内容的相关性、准确性和连贯性。

ATLANTIC: 结构感知的检索增强语言模型用于跨学科科学

ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for  Interdisciplinary Science

In recent years, Massive Open Online Courses (MOOCs) have gained significant
traction as a rapidly growing phenomenon in online learning. Unlike traditional
classrooms, MOOCs offer a unique opportunity to cater to a diverse audience
from different backgrounds and geographical locations. Renowned universities
and MOOC-specific providers, such as Coursera, offer MOOC courses on various
subjects. Automated assessment tasks like grade and early dropout predictions
are necessary due to the high enrollment and limited direct interaction between
teachers and learners. However, current automated assessment approaches
overlook the structural links between different entities involved in the
downstream tasks, such as the students and courses. Our hypothesis suggests
that these structural relationships, manifested through an interaction graph,
contain valuable information that can enhance the performance of the task at
hand. To validate this, we construct a unique knowledge graph for a large MOOC
dataset, which will be publicly available to the research community.
Furthermore, we utilize graph embedding techniques to extract latent structural
information encoded in the interactions between entities in the dataset. These
techniques do not require ground truth labels and can be utilized for various
tasks. Finally, by combining entity-specific features, behavioral features, and
extracted structural features, we enhance the performance of predictive machine
learning models in student assignment grade prediction. Our experiments
demonstrate that structural features can significantly improve the predictive
performance of downstream assessment tasks. The code and data are available in
https://github.com/DSAatUSU/MOOPer_grade_prediction

利用大规模在线开放课程（MOOC）数据集构建独特的知识图谱，并运用图嵌入技术提取实体间相互作用中潜在结构信息，结合实体特征、行为特征和结构特征，改进了对学生成绩的预测模型的性能。

利用图表示学习提升 MOOC 自动等级预测性能

Enhancing the Performance of Automated Grade Prediction in MOOC using  Graph Representation Learning

Node classification tasks on graphs are addressed via fully-trained deep
message-passing models that learn a hierarchy of node representations via
multiple aggregations of a node's neighbourhood. While effective on graphs that
exhibit a high ratio of intra-class edges, this approach poses challenges in
the opposite case, i.e. heterophily, where nodes belonging to the same class
are usually further apart. In graphs with a high degree of heterophily, the
smoothed representations based on close neighbours computed by convolutional
models are no longer effective. So far, architectural variations in
message-passing models to reduce excessive smoothing or rewiring the input
graph to improve longer-range message passing have been proposed. In this
paper, we address the challenges of heterophilic graphs with Graph Echo State
Network (GESN) for node classification. GESN is a reservoir computing model for
graphs, where node embeddings are recursively computed by an untrained
message-passing function. Our experiments show that reservoir models are able
to achieve better or comparable accuracy with respect to most fully trained
deep models that implement ad hoc variations in the architectural bias or
perform rewiring as a preprocessing step on the input graph, with an
improvement in terms of efficiency/accuracy trade-off. Furthermore, our
analysis shows that GESN is able to effectively encode the structural
relationships of a graph node, by showing a correlation between iterations of
the recursive embedding function and the distribution of shortest paths in a
graph.

在异质性图上处理节点分类任务是具有挑战性的，本文提出使用图回声状态网络（Graph Echo State Network，GESN）来解决这一挑战。结果表明，与完全训练的深度模型相比，GESN 能够实现更好或相当的精度，同时在效率 / 准确性平衡方面有所改进，并且能够有效编码图节点的结构关系。