Affiliations: [a] School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, P.R. China
| [b] School of Information Science and Technology, Zhongkai University of Agriculture and Engineering, Guangzhou, P.R. China
Correspondence:
[*]
Corresponding author. Huiting Hu, School of Information Science and Technology, Zhongkai University of Agriculture and Engineering, Guangzhou, P.R. China. Tel: +86 13826279352; E-mail: huhuit0105@gmail.com.
Abstract: In recent mathematical reasoning tasks, self-attention has achieved better results in public datasets. However, self-attention performs poorly on more complex mathematical problems due to the lack of capacity to capture local features and the ill-conditioned training after deepening the number of layers. To tackle the problem and enhance its ability of extracting local features while learning the global contexts, we propose an implicit mathematical reasoning model that improves Transformer by combining self-attention and convolution to achieve joint modeling of global and local context. Also, by introducing Reweight connection and adversarial loss function, we prevent the model gradient from disappearing or exploding in a deep neural network while ensuring the convergence speed and avoiding overfitting. Experimental results show that the proposed model improves the accuracy by 4.47% on average for complex mathematical problems compared to the best existing results. In addition, we verify the validity of our model using ablation analysis and further demonstrate the interpretability of the model by attention mapping and task role analysis.
Keywords: Implicit mathematical reasoning, self-attention, depth separable convolution, causal language model, adversarial loss
