Knowledge enhancement BERT based on domain dictionary mask

Article type: Research Article

Authors: Cao, Xianglin | Xiao, Hong | Jiang, Wenchao^{; *}

Affiliations: School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China

Correspondence: [*] Corresponding author. E-mail: jiangwenchao@gdut.edu.cn.

Abstract: Semantic matching is one of the critical technologies for intelligent customer service. Since Bidirectional Encoder Representations from Transformers (BERT) is proposed, fine-tuning on a large-scale pre-training language model becomes a general method to implement text semantic matching. However, in practical application, the accuracy of the BERT model is limited by the quantity of pre-training corpus and proper nouns in the target domain. An enhancement method for knowledge based on domain dictionary to mask input is proposed to solve the problem. Firstly, for modul input, we use keyword matching to recognize and mask the word in domain. Secondly, using self-supervised learning to inject knowledge of the target domain into the BERT model. Thirdly, we fine-tune the BERT model with public datasets LCQMC and BQboost. Finally, we test the model’s performance with a financial company’s user data. The experimental results show that after using our method and BQboost, accuracy increases by 12.12% on average in practical applications.

Keywords: Intelligent customer service, dictionary mask, BERT, data preprocessing

DOI: 10.3233/JHS-222013

Journal: Journal of High Speed Networks, vol. 29, no. 2, pp. 121-128, 2023