Evaluating Robustness of LLMs to Numerical Variations in Mathematical Reasoning
Published in 言語処理学会第31回年次大会発表論文集, 2025
Abstract
Evaluating an LLM’s robustness against numerical perturbation is a good way to know if the LLM actually performs reasoning or just replicates patterns learned. We propose a novel method to augment math word problems (MWPs), producing numerical variations at a large scale utilizing templates. We also propose an automated error classification framework for scalable error analysis, distinguishing calculation errors from reasoning errors. Our experiments using the methods show LLMs are weak against numerical variations, suggesting they are not fully capable of generating valid reasoning steps, often failing in arithmetic operations.
Recommended citation:
Yang Yuli，山田寛章，徳永健伸. Evaluating Robustness of LLMs to Numerical Variations in Mathematical Reasoning. 言語処理学会第31回年次大会発表論文集, pp. 851-856，2025年3月.