Augment BERT with average pooling layer for Chinese summary generation

Article type: Research Article

Authors: Zhao, Shuai^{a; b; *} | You, Fucheng^b | Chang, Wen^b | Zhang, Tianyu^b | Hu, Man^b

Affiliations: [a] Jinan University, Guangzhou, China | [b] Beijing Institute of Graphic Communication, Beijing, China

Correspondence: [*] Corresponding author. Shuai Zhao. E-mail: 17839192463@163.com.

Abstract: The BERT pre-trained language model has achieved good results in various subtasks of natural language processing, but its performance in generating Chinese summaries is not ideal. The most intuitive reason is that the BERT model is based on character-level composition, while the Chinese language is mostly in the form of phrases. Directly fine-tuning the BERT model cannot achieve the expected effect. This paper proposes a novel summary generation model with BERT augmented by the pooling layer. In our model, we perform an average pooling operation on token embedding to improve the model’s ability to capture phrase-level semantic information. We use LCSTS and NLPCC2017 to verify our proposed method. Experimental data shows that the average pooling model’s introduction can effectively improve the generated summary quality. Furthermore, different data needs to be set with varying pooling kernel sizes to achieve the best results through comparative analysis. In addition, our proposed method has strong generalizability. It can be applied not only to the task of generating summaries, but also to other natural language processing tasks.

Keywords: Summary generation, fine-tuning bert, average pooling, transformer

DOI: 10.3233/JIFS-211229

Journal: Journal of Intelligent & Fuzzy Systems, vol. 42, no. 3, pp. 1859-1868, 2022