A large majority of American adults get at least some of their news from the
Internet. Even though many online news products have the goal of informing
their users about the news, they lack scalable and reliable tools for measuring
how well they are achieving this goal, and therefore have to resort to noisy
proxy metrics (e.g., click-through rates or reading time) to track their
performance.
As a first step towards measuring news informedness at a scale, we study the
problem of quiz-style multiple-choice question generation, which may be used to
survey users about their knowledge of recent news. In particular, we formulate
the problem as two sequence-to-sequence tasks: question-answer generation (QAG)
and distractor, or incorrect answer, generation (DG). We introduce NewsQuizQA,
the first dataset intended for quiz-style question-answer generation,
containing 20K human written question-answer pairs from 5K news article
summaries. Using this dataset, we propose a series of novel techniques for
applying large pre-trained Transformer encoder-decoder models, namely PEGASUS
and T5, to the tasks of question-answer generation and distractor generation.
We show that our models outperform strong baselines using both automated
metrics and human raters. We provide a case study of running weekly quizzes on
real-world users via the Google Surveys platform over the course of two months.
We found that users generally found the automatically generated questions to be
educational and enjoyable. Finally, to serve the research community, we are
releasing the NewsQuizQA dataset.

本研究通过建立一套问答生成模型，实现了对在线新闻读者对新闻信息的认知度的度量，并通过实验证明该模型在多项指标上具有优越的性能。