Affiliations: [a] School of Information Engineering, Shenyang University of Chemical Technology, shenyang, China
| [b] State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China
| [c] School of Electronics and Communication Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
| [d] Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China
Note: [1] Yimin Zhu and Qing Gao contributed equally to this work.
Abstract: Gestures have long been recognized as an interaction technique that can provide a more natural, creative, and intuitive way to communicate with computers. However, some existing difficulties include the high probability that the same type of movement done at different speeds will be recognized as a different category of movement; cluttered, occluded, and low-resolution backgrounds; and the near-impossibility of fusing different types of features. To this end, we propose a novel framework for integrating different scales of RGB and motion skeletons to obtain higher recognition accuracy using multiple features. Specifically, we provide a network architecture that combines a three-dimensional convolutional neural network (3DCNN) and post-fusion to better embed different features. Also, we combine RGB and motion skeleton information at different scales to mitigate speed and background issues. Experiments on several gesture recognition public datasets show desirable results, validating the superiority of the proposed gesture recognition method. Finally, we do a human-computer interaction experiment to prove its practicality.
Keywords: Multi-modal action recognition, body action, robot simulation
