The increasing use of transformer-based large language models brings forward
the challenge of processing long sequences. In document visual question
answering (DocVQA), leading methods focus on the single-page setting, while
documents can span hundreds of pages. We present GRAM, a method that seamlessly
extends pre-trained single-page models to the multi-page setting, without
requiring computationally-heavy pretraining. To do so, we leverage a
single-page encoder for local page-level understanding, and enhance it with
document-level designated layers and learnable tokens, facilitating the flow of
information across pages for global reasoning. To enforce our model to utilize
the newly introduced document-level tokens, we propose a tailored bias
adaptation method. For additional computational savings during decoding, we
introduce an optional compression stage using our C-Former model, which reduces
the encoded sequence length, thereby allowing a tradeoff between quality and
latency. Extensive experiments showcase GRAM's state-of-the-art performance on
the benchmarks for multi-page DocVQA, demonstrating the effectiveness of our
approach.

在多页文档问答中，我们提出了 GRAM 方法，它能够无需复杂的预训练，将单页模型扩展到多页设置，并通过引入文档级指定层和可学习令牌来增强本地页面级理解，促进跨页面信息流动。大量实验表明 GRAM 在多页文档问答基准测试中具有先进的性能，展示了我们方法的有效性。

GRAM: 跨页面多模态问答的全局推理

GRAM: Global Reasoning for Multi-Page VQA

Content-based collaborative filtering (CCF) predicts user-item interactions
based on both users' interaction history and items' content information.
Recently, pre-trained language models (PLM) have been used to extract
high-quality item encodings for CCF. However, it is resource-intensive to train
a PLM-based CCF model in an end-to-end (E2E) manner, since optimization
involves back-propagating through every content encoding within a given user
interaction sequence. To tackle this issue, we propose GRAM (GRadient
Accumulation for Multi-modality in CCF), which exploits the fact that a given
item often appears multiple times within a batch of interaction histories.
Specifically, Single-step GRAM aggregates each item encoding's gradients for
back-propagation, with theoretic equivalence to the standard E2E training. As
an extension of Single-step GRAM, we propose Multi-step GRAM, which increases
the gradient update latency, achieving a further speedup with drastically less
GPU memory. GRAM significantly improves training efficiency (up to 146x) on
five datasets from two task domains of Knowledge Tracing and News
Recommendation. Our code is available at this https URL.

提出了 GRAM（CCF 中的 GRadient Accumulation for Multi-modality），它利用了同一批交互历史记录中单个物品经常出现多次的事实来解决通过预训练语言模型提取高质量物品编码的资源密集型问题，显著提高了基于五个数据集的任务领域（知识追踪和新闻推荐）中的训练效率（最大提升 146 倍），Single-step GRAM 和 Multi-step GRAM 分别是 GRAM 的一个单步版本和扩展版本。

GRAM: 面向内容的协同过滤的快速预训练语言模型微调

GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering

Graphs are ubiquitous real-world data structures, and generative models that
approximate distributions over graphs and derive new samples from them have
significant importance. Among the known challenges in graph generation tasks,
scalability handling of large graphs and datasets is one of the most important
for practical applications. Recently, an increasing number of graph generative
models have been proposed and have demonstrated impressive results. However,
scalability is still an unresolved problem due to the complex generation
process or difficulty in training parallelization. In this paper, we first
define scalability from three different perspectives: number of nodes, data,
and node/edge labels. Then, we propose GRAM, a generative model for graphs that
is scalable in all three contexts, especially in training. We aim to achieve
scalability by employing a novel graph attention mechanism, formulating the
likelihood of graphs in a simple and general manner. Also, we apply two
techniques to reduce computational complexity. Furthermore, we construct a
unified and non-domain-specific evaluation metric in node/edge-labeled graph
generation tasks by combining a graph kernel and Maximum Mean Discrepancy. Our
experiments on synthetic and real-world graphs demonstrated the scalability of
our models and their superior performance compared with baseline methods.

本文介绍了一种可扩展的图生成模型 GRAM，其采用一种新颖的图注意机制和评估方法，在节点 / 边标签图生成任务中表现出较高的性能。