Recent literature has underscored the importance of dataset documentation
work for machine learning, and part of this work involves addressing
"documentation debt" for datasets that have been used widely but documented
sparsely. This paper aims to help address documentation debt for BookCorpus, a
popular text dataset for training large language models. Notably, researchers
have used BookCorpus to train OpenAI's GPT-N models and Google's BERT models,
even though little to no documentation exists about the dataset's motivation,
composition, collection process, etc. We offer a preliminary datasheet that
provides key context and information about BookCorpus, highlighting several
notable deficiencies. In particular, we find evidence that (1) BookCorpus
likely violates copyright restrictions for many books, (2) BookCorpus contains
thousands of duplicated books, and (3) BookCorpus exhibits significant skews in
genre representation. We also find hints of other potential deficiencies that
call for future research, including problematic content, potential skews in
religious representation, and lopsided author contributions. While more work
remains, this initial effort to provide a datasheet for BookCorpus adds to
growing literature that urges more careful and systematic documentation for
machine learning datasets.

本文旨在帮助解决 BookCorpus 数据集的文档债务问题，提供初步数据表，揭示了该数据集存在违反版权限制、大量重复书籍以及流派偏差等问题，并呼吁更加注意和系统化的机器学习数据集文档化工作。