mathematical reasoning is regarded as a necessary ability for language models (LMs). Recent works demonstrate large LMs' impressive performance in solving math problems. The success is attributed to their Chain-o