Existing full text datasets of U.S. public domain newspapers do not recognize
the often complex layouts of newspaper scans, and as a result the digitized
content scrambles texts from articles, headlines, captions, advertisements, and
other layout regions. OCR quality can also be low. This study develops a novel,
deep learning pipeline for extracting full article texts from newspaper images
and applies it to the nearly 20 million scans in Library of Congress's public
domain Chronicling America collection. The pipeline includes layout detection,
legibility classification, custom OCR, and association of article texts
spanning multiple bounding boxes. To achieve high scalability, it is built with
efficient architectures designed for mobile phones. The resulting American
Stories dataset provides high quality data that could be used for pre-training
a large language model to achieve better understanding of historical English
and historical world knowledge. The dataset could also be added to the external
database of a retrieval-augmented language model to make historical information
- ranging from interpretations of political events to minutiae about the lives
of people's ancestors - more widely accessible. Furthermore, structured article
texts facilitate using transformer-based methods for popular social science
applications like topic classification, detection of reproduced content, and
news story clustering. Finally, American Stories provides a massive silver
quality dataset for innovating multimodal layout analysis models and other
multimodal applications.

用深度学习方法从报纸图像中提取全文文章，以构建高质量数据集供大型语言模型预训练，达到对历史英语和历史世界知识有更好理解的目的。