While (large) language models have significantly improved over the last
years, they still struggle to sensibly process long sequences found, e.g., in
books, due to the quadratic scaling of the underlying attention mechanism. To
address this, we propose NextLevelBERT, a Masked Language Model operating not
on tokens, but on higher-level semantic representations in the form of text
embeddings. We pretrain NextLevelBERT to predict the vector representation of
entire masked text chunks and evaluate the effectiveness of the resulting
document vectors on three task types: 1) Semantic Textual Similarity via
zero-shot document embeddings, 2) Long document classification, 3)
Multiple-choice question answering. We find that next level Masked Language
Modeling is an effective technique to tackle long-document use cases and can
outperform much larger embedding models as long as the required level of detail
is not too high. We make model and code available.

NextLevelBERT 是一种掩码语言模型，通过对高级语义表示进行预测，可以有效处理长文档，并在语义文本相似性、长文档分类和多项选择问答三个任务类型上取得较好效果。

NextLevelBERT：探索使用高级表示进行长文档的遮蔽语言建模

NextLevelBERT: Investigating Masked Language Modeling with Higher-Level  Representations for Long Documents

Text classification aims to assign labels to textual units by making use of
global information. Recent studies have applied graph neural network (GNN) to
capture the global word co-occurrence in a corpus. Existing approaches require
that all the nodes (training and test) in a graph are present during training,
which are transductive and do not naturally generalise to unseen nodes. To make
those models inductive, they use extra resources, like pretrained word
embedding. However, high-quality resource is not always available and hard to
train. Under the extreme settings with no extra resource and limited amount of
training set, can we still learn an inductive graph-based text classification
model? In this paper, we introduce a novel inductive graph-based text
classification framework, InducT-GCN (InducTive Graph Convolutional Networks
for Text classification). Compared to transductive models that require test
documents in training, we construct a graph based on the statistics of training
documents only and represent document vectors with a weighted sum of word
vectors. We then conduct one-directional GCN propagation during testing. Across
five text classification benchmarks, our InducT-GCN outperformed
state-of-the-art methods that are either transductive in nature or pre-trained
additional resources. We also conducted scalability testing by gradually
increasing the data size and revealed that our InducT-GCN can reduce the time
and space complexity. The code is available on:
this https URL.

本研究提出了一种新颖的归纳图卷积网络框架 InducT-GCN（InducTive Graph Convolutional Networks for Text classification），用于无额外资源和有限训练数据情况下的归纳图文本分类，通过仅基于训练文档的统计信息构建图并在测试期间进行单向 GCN 传播，与传统感性学习模型相比，InducT-GCN 在五个文本分类基准测试中表现出更好的性能。