Quantitative and numerical comprehension in language is an important task in
many fields like education and finance, but still remains a challenging task
for language models. While tool and calculator usage has shown to be helpful to
improve mathematical reasoning in large pretrained decoder-only language
models, this remains unexplored for smaller language models with encoders. In
this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning
to use the calculator for both encoder-only and encoder-decoder architectures,
formulated as a discriminative and generative task respectively. We pre-train
BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative
calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves
performance on downstream tasks that require numerical understanding. Our code
and data are available at this https URL

利用计算器对较小的编码器型语言模型进行预微调，可以提高在需要数值理解的下游任务中的性能。