Unstructured data, especially text, continues to grow rapidly in various
domains. In particular, in the financial sphere, there is a wealth of
accumulated unstructured financial data, such as the textual disclosure
documents that companies submit on a regular basis to regulatory agencies, such
as the Securities and Exchange Commission (SEC). These documents are typically
very long and tend to contain valuable soft information about a company's
performance. It is therefore of great interest to learn predictive models from
these long textual documents, especially for forecasting numerical key
performance indicators (KPIs). Whereas there has been a great progress in
pre-trained language models (LMs) that learn from tremendously large corpora of
textual data, they still struggle in terms of effective representations for
long documents. Our work fills this critical need, namely how to develop better
models to extract useful information from long textual documents and learn
effective features that can leverage the soft financial and risk information
for text regression (prediction) tasks. In this paper, we propose and implement
a deep learning framework that splits long documents into chunks and utilizes
pre-trained LMs to process and aggregate the chunks into vector
representations, followed by self-attention to extract valuable document-level
features. We evaluate our model on a collection of 10-K public disclosure
reports from US banks, and another dataset of reports submitted by US
companies. Overall, our framework outperforms strong baseline methods for
textual modeling as well as a baseline regression model using only numerical
data. Our work provides better insights into how utilizing pre-trained
domain-specific and fine-tuned long-input LMs in representing long documents
can improve the quality of representation of textual data, and therefore, help
in improving predictive analyses.

我们提出并实现了一个利用预训练语言模型处理长文本文档的深度学习框架，将长文档分成块，并利用自注意力机制提取有价值的文档级特征，该框架在美国银行的 10-K 公开披露报告和其他美国公司提交的报告数据集上表现出色，是文本建模和回归模型的更好选择，利用预训练的领域特定和微调的模型在表示文本数据方面的效果较好，从而有助于提高预测分析的质量。

FETILDA：长篇财经文本嵌入的有效调优框架

FETILDA: An Effective Framework For Fin-tuned Embeddings For Long Financial Text Documents

Current state-of-the-art approaches for named entity recognition (NER)
typically consider text at the sentence-level and thus do not model information
that crosses sentence boundaries. However, the use of transformer-based models
for NER offers natural options for capturing document-level features. In this
paper, we perform a comparative evaluation of document-level features in the
two standard NER architectures commonly considered in the literature, namely
"fine-tuning" and "feature-based LSTM-CRF". We evaluate different
hyperparameters for document-level features such as context window size and
enforcing document-locality. We present experiments from which we derive
recommendations for how to model document context and present new
state-of-the-art scores on several CoNLL-03 benchmark datasets. Our approach is
integrated into the Flair framework to facilitate reproduction of our
experiments.

本文针对不考虑跨句子边界信息的标准命名实体识别（NER）模型，通过利用 Transformer 模型进行文档级别特征抽取，提出了基于 Fine-tuning 和 LSTM-CRF 的文档级特征 NER 模型，并通过对上下文窗口大小和文档本地性等不同超参数的实验，得出如何建模文档上下文的结论，最后在多个 CoNLL-03 基准数据集上验证提出的方法取得了最新的最优结果。

FLERT：用于命名实体识别的文档级特征

FLERT: Document-Level Features for Named Entity Recognition

Text summarization aims to compress a textual document to a short summary
while keeping salient information. Extractive approaches are widely used in
text summarization because of their fluency and efficiency. However, most of
existing extractive models hardly capture inter-sentence relationships,
particularly in long documents. They also often ignore the effect of topical
information on capturing important contents. To address these issues, this
paper proposes a graph neural network (GNN)-based extractive summarization
model, enabling to capture inter-sentence relationships efficiently via
graph-structured document representation. Moreover, our model integrates a
joint neural topic model (NTM) to discover latent topics, which can provide
document-level features for sentence selection. The experimental results
demonstrate that our model not only substantially achieves state-of-the-art
results on CNN/DM and NYT datasets but also considerably outperforms existing
approaches on scientific paper datasets consisting of much longer documents,
indicating its better robustness in document genres and lengths. Further
discussions show that topical information can help the model preselect salient
contents from an entire document, which interprets its effectiveness in long
document summarization.

本文提出了基于图神经网络（GNN）的抽取式摘要模型，并整合了深度神经主题模型（NTM）来发现潜在主题，从而提供句子选择的文档级特征，实验结果表明，我们的模型在 CNN/DM 和 NYT 数据集上取得了最先进的结果，在由较长文档组成的科学论文数据集上也明显优于现有方法，进一步研究表明，主题信息可以帮助模型从整个文档中预先选择凸显内容，因此能够有效地摘要长文档。