This paper introduces Fundus, a user-friendly news scraper that enables users
to obtain millions of high-quality news articles with just a few lines of code.
Unlike existing news scrapers, we use manually crafted, bespoke content
extractors that are specifically tailored to the formatting guidelines of each
supported online newspaper. This allows us to optimize our scraping for quality
such that retrieved news articles are textually complete and without HTML
artifacts. Further, our framework combines both crawling (retrieving HTML from
the web or large web archives) and content extraction into a single pipeline.
By providing a unified interface for a predefined collection of newspapers, we
aim to make Fundus broadly usable even for non-technical users. This paper
gives an overview of the framework, discusses our design choices, and presents
a comparative evaluation against other popular news scrapers. Our evaluation
shows that Fundus yields significantly higher quality extractions (complete and
artifact-free news articles) than prior work. The framework is available on
GitHub under this https URL and can be simply installed
using pip.

该论文介绍了 Fundus，一种用户友好的新闻爬取工具，用户只需几行代码即可获取数百万篇高质量的新闻文章。与现有的新闻爬取工具不同，我们使用手工制作的定制内容抽取器，专门适应每个支持的在线报纸的格式指南，从而优化我们的爬取质量，以获取完整且无 HTML 残留物的新闻文章。此外，我们的框架将抓取（从网络或大型网络档案中检索 HTML）和内容抽取合并为一个单一流程。通过为预定义的报纸集合提供统一接口，我们旨在使 Fundus 即使对非技术用户也广泛可用。该论文概述了该框架，讨论了我们的设计选择，并与其他流行的新闻爬取工具进行了比较评估。我们的评估结果显示，与以往的工作相比，Fundus 能够产生具有显著更高质量的抽取结果（完整且无残留的新闻文章）。该框架可以在 GitHub 的此 https URL 下获得，并可通过 pip 轻松安装。