FPGA-based design and implementation of the location attention mechanism in neural networks

Article type: Research Article

Authors: Qiao, Ruixiu^{a; b; 1} | Guo, Xiaozhou^{a; b; 1} | Mao, Wenyu^{a; b; *} | Li, Jixing^{a; b} | Lu, Huaxiang^{a; b; c; d}

Affiliations: [a] Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China | [b] University of Chinese Academy of Sciences, Beijing, China | [c] CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Beijing, China | [d] Semiconductor Neural Network Intelligent Perception and Computing Technology Beijing Key Lab, Beijing, China

Correspondence: [*] Corresponding author. Wenyu Mao, E-mail: maowenyu@semi.ac.cn.

Note: [1] These authors contributed equally to this work.

Abstract: The location attention mechanism has been widely applied in deep neural networks. However, as the mechanism entails heavy computing workload, significant memories consumed for weights storage, and shows poor parallelism in some calculations, it is hard to achieve high efficiency deployment. In this paper, the field-programmable gate array (FPGA) is employed to implement the location attention mechanism in hardware, and a novel fusion approach is proposed to connect the convolutional layer with the fully connected layer, which not only improves the parallelism of both the algorithm and the hardware pipeline, but also reduces the computation cost for such operations as multiplication and addition. Meanwhile, the shared computing architecture is used to reduce the demand for hardware resources. Parallel computing arrays are utilized to time-multiplex a single computing array, which can speed up the pipeline parallel computing of the attention mechanism. Experimental results show that for the location attention mechanism, the FPGA’s inference speed is 0.010 ms, which is around a quarter of the speed achieved by running it with GPU, and its power consumption is 1.73 W, which is about 2.89% of the power consumed by running it with CPU. Compared with other FPGA implementation methods of attention mechanism, it has less hardware resource consumption and less inference time. When applied to speech recognition tasks, the trained attention model is symmetrically quantized and deployed on the FPGA. The result shows that the word error rate is only 0.79% higher than that before quantization, which proves the effectiveness and correctness of the hardware circuit.

Keywords: Attention mechanism, neural networks, FPGA, deep learning, hardware implementation

DOI: 10.3233/JIFS-212273

Journal: Journal of Intelligent & Fuzzy Systems, vol. 43, no. 4, pp. 5309-5323, 2022