Abstractive text summarization has recently become a popular approach, but
data hallucination remains a serious problem, including with quantitative data.
We propose a set of probing tests to evaluate the efficacy of abstract
summarization models' modeling of quantitative values found in the input text.
Our results show that in most cases, the encoders of recent SOTA-performing
models struggle to provide embeddings that adequately represent quantitative
values in the input compared to baselines, and in particular, they outperform
random representations in some, but surprisingly not all, cases. Under our
assumptions, this suggests that the encoder's performance contributes to the
quantity hallucination problem. One model type in particular, DistilBART-CDM,
was observed to underperform randomly initialized representations for several
experiments, and performance versus BERT suggests that standard pretraining and
fine-tuning approaches for the summarization task may play a role in
underperformance for some encoders.

本文提出了一套探测测试用于评估抽象自动摘要模型中建模输入文本中定量价值的有效性，并发现 DistilBART-CDM 模型在多个实验中表现不佳，并且与 BERT 的性能比较表明，摘要任务的标准预训练和微调方法可能对一些编码器的表现产生影响。