Dual-cell differentiable architecture search for language modeling

Article type: Research Article

Authors: Wan, Quan^{a; b} | Wu, Lin^{a; b; *} | Yu, Zhengtao^{a; b}

Affiliations: [a] Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China | [b] Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming, China

Correspondence: [*] Corresponding author. Lin Wu, Tel.: +86 158 0876 7695; E-mail: 52038994@qq.com.

Abstract: Initial results of neural architecture search (NAS) in natural language processing (NLP) have been achieved, but the search space of most NAS methods is based on the simplest recurrent cell and thus does not consider the modeling of long sequences. The remote information tends to disappear gradually when the input sequence is long, resulting in poor model performance. In this paper, we present an approach based on dual cells to search for a better-performing network architecture. We construct a search space that is more compatible with language modeling tasks by adding an information storage cell inside the search cell, so that we can make better use of the remote information of the sequence and improve the performance of the model. The language model searched by our method achieves better results than those of the baseline method on the Penn Treebank data set and WikiText-2 data set.

Keywords: Neural architecture search, natural language processing, recurrent neural network

DOI: 10.3233/JIFS-210207

Journal: Journal of Intelligent & Fuzzy Systems, vol. 41, no. 2, pp. 3985-3992, 2021