Seq2seq model for human action recognition based on skeleton and two-layer bidirectional LSTM

Article type: Research Article

Authors: Wei, Shouke^{a; b; c} | Zhao, Jindong^{b; *} | Li, Junhuai^a | Yuan, Meixue^b

Affiliations: [a] School of Computer Science and Engineering, Xi’an University of Technology, Xi’an, China | [b] School of Computer and Control Engineering, Yantai University, Yantai, China | [c] Deepsim Intelligence Technology Inc., Abbotsford, BC, Canada

Correspondence: [*] Corresponding author. E-mail: zhjdong@ytu.edu.cn.

Abstract: Human action recognition (HAR) plays an important role in social interaction in various fields. This study proposes a light-weight skeleton and two-layer bidirectional LSTM-based Seq2Seq model (SB2_Seq2Seq) for HAR to trade off recognition accuracy, users’ privacy and computer resource usage. An experiment was conducted to compare the proposed SB2_Seq2Seq with other skeleton-based Seq2Seq models and non-skeleton RGB video frame-based LSTM, CNN and seq2seq models. The UCF50 dataset was used for model evaluation, where 60%, 20% and 20% for model training, validation and testing, respectively. The experimental results show that the proposed model achieves 93.54% accuracy with 0.0214 Mean Square Error (MSE), suggesting that the proposed model outperforms all the other models. Besides, it also shows that the proposed model achieves state-of-the-art accuracy compared with state-of-the-arts methods in literature.

Keywords: Human action recognition (HAR), skeleton, Seq2Seq, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), deep leaning, tradeoff, privacy, UCF50

DOI: 10.3233/AIS-220125

Journal: Journal of Ambient Intelligence and Smart Environments, vol. 15, no. 4, pp. 315-331, 2023