Affiliations: [a] Center for Applied Mathematics, Tianjin University, Tianjin, China | [b] Institute of Automation, Chinese Academy of Sciences, Beijing, China
Correspondence:
[*]
Corresponding author: Ou Wu, Center for Applied Mathematics, Tianjin University, Tianjin, China. E-mail: wuou@tju.edu.cn.
Abstract: Text sentiment analysis is an important natural language processing (NLP) task and has received considerable attention in recent years. Numerous deep-learning based methods have been proposed in previous literature in terms of new deep neural networks (DNN) including new embedding strategies, new attention mechanisms, and new encoding layers. In this study, an alternative technical path is investigated to further improve the state-of-the-art performance of text sentiment analysis. An new effective learning framework is proposed that combines knowledge distillation and sample selection. A dually-born-again network (DBAN) is presented in which the teacher network and the student network are simultaneously trained through an iterative approach. A selection gate is defined to deal with training samples which are useless or even harmful for model training. Moreover, both the DBAN and sample selection are further improved by ensemble. The proposed framework can improve the existing state-of-the-art DNN models in sentiment analysis. Experimental results indicate that the proposed framework enhances the performances of existing networks. In addition, DBAN outperforms existing born-again network.
Keywords: Classification, deep neural network, knowledge distillation, sample selection
