Localization and classification of human facial emotions using local intensity order pattern and shape-based texture features

Article type: Research Article

Authors: Kalsum, Tehmina^a | Mehmood, Zahid^{b; *} | Kulsoom, Farzana^c | Chaudhry, Hassan Nazeer^d | Khan, Amjad Rehman^e | Rashid, Muhammad^f | Saba, Tanzila^e

Affiliations: [a] Department of Software Engineering, University of Engineering and Technology Taxila, Taxila, Pakistan | [b] Department of Computer Engineering, University of Engineering and Technology Taxila, Taxila, Pakistan | [c] Department of Electrical, Computer, and Biomedical Engineering, University of Pavia, Pavia, Itlay | [d] Department of Electrical, Information, and Bio Engineering, Politecnico di Milano, Milan, Itlay | [e] Artificial Intelligence & Data Analytics (AIDA) Lab, CCIS, Prince Sultan University, Riyadh, Saudi Arabia | [f] Department of Computer Engineering, Umm Al-Qura University, Makkah, Saudi Arabia

Correspondence: [*] Corresponding author. Zahid Mehmood, Department of Computer Engineering, University of Engineering and Technology Taxila, Taxila, Pakistan. E-mail: zahid.mehmood@uettaxila.edu.pk.

Abstract: Facial emotion recognition system (FERS) recognize the person’s emotions based on various image processing stages including feature extraction as one of the major processing steps. In this study, we presented a hybrid approach for recognizing facial expressions by performing the feature level fusion of a local and a global feature descriptor that is classified by a support vector machine (SVM) classifier. Histogram of oriented gradients (HoG) is selected for the extraction of global facial features and local intensity order pattern (LIOP) to extract the local features. As HoG is a shape-based descriptor, with the help of edge information, it can extract the deformations caused in facial muscles due to changing emotions. On the contrary, LIOP works based on the information of pixels intensity order and is invariant to change in image viewpoint, illumination conditions, JPEG compression, and image blurring as well. Thus both the descriptors proved useful to recognize the emotions effectively in the images captured in both constrained and realistic scenarios. The performance of the proposed model is evaluated based on the lab-constrained datasets including CK+, TFEID, JAFFE as well as on realistic datasets including SFEW, RaF, and FER-2013 dataset. The optimal recognition accuracy of 99.8%, 98.2%, 93.5%, 78.1%, 63.0%, 56.0% achieved respectively for CK+, JAFFE, TFEID, RaF, FER-2013 and SFEW datasets respectively.

Keywords: Facial emotion recognition, histogram-of-oriented-gradients, local intensity order pattern, support vector machine, texture features

DOI: 10.3233/JIFS-201799

Journal: Journal of Intelligent & Fuzzy Systems, vol. 40, no. 5, pp. 9311-9331, 2021