Over the past few years, various domain-specific pretrained language models
(PLMs) have been proposed and have outperformed general-domain PLMs in
specialized areas such as biomedical, scientific, and clinical domains. In
addition, financial PLMs have been studied because of the high economic impact
of financial data analysis. However, we found that financial PLMs were not
pretrained on sufficiently diverse financial data. This lack of diverse
training data leads to a subpar generalization performance, resulting in
general-purpose PLMs, including BERT, often outperforming financial PLMs on
many downstream tasks. To address this issue, we collected a broad range of
financial corpus and trained the Financial Language Model (FiLM) on these
diverse datasets. Our experimental results confirm that FiLM outperforms not
only existing financial PLMs but also general domain PLMs. Furthermore, we
provide empirical evidence that this improvement can be achieved even for
unseen corpus groups.

过去几年中，已经提出了各种领域特定的预训练语言模型（PLMs），并在生物医学、科学和临床领域等专业领域表现优于通用领域的 PLMs。此外，由于财务数据分析的经济影响巨大，金融 PLMs 也受到了研究的关注。然而，我们发现金融 PLMs 在预训练过程中未使用足够多样化的金融数据。这种缺乏多样化训练数据导致其泛化性能不佳，导致通用 PLMs（包括 BERT）在许多下游任务上通常胜过金融 PLMs。为了解决这个问题，我们收集了广泛的金融语料库，并在这些多样化的数据集上训练了金融语言模型（FiLM）。我们的实验结果证实，FiLM 不仅在现有金融 PLMs 上表现出色，还在通用领域 PLMs 上表现优异。此外，我们提供了实证证据，即这种改进甚至适用于未见过的语料库组。