Video representation and suspicious event detection using semantic technologies

Issue title: Open Science Data and the Semantic Web Journal

Article type: Research Article

Authors: Patel, Ashish Singh^{a; *} | Merlino, Giovanni^b | Bruneo, Dario^b | Puliafito, Antonio^b | Vyas, O.P.^c | Ojha, Muneendra^a

Affiliations: [a] Department of Computer Science and Engineering, DSPM International Institute of Information Technology Naya Raipur, Atal Nagar, Raipur, India. E-mails: ashish@iiitnr.edu.in, muneendra@iiitnr.edu.in | [b] Department of Engineering, University of Messina, Messina, Italy. E-mails: gmerlino@unime.it, dbruneo@unime.it, apuliafito@unime.it | [c] Department of Information Technology, Indian Institute of Information Technology-Allahabad, Prayagraj, India. E-mail: dropvyas@gmail.com

Correspondence: [*] Corresponding author. E-mail: ashish@iiitnr.edu.in.

Abstract: Storage and analysis of video surveillance data is a significant challenge, requiring video interpretation and event detection in the relevant context. To perform this task, the low-level features including shape, texture, and color information are extracted and represented in symbolic forms. In this work, a methodology is proposed, which extracts the salient features and properties using machine learning techniques and represent this information as Linked Data using a domain ontology that is explicitly tailored for detection of certain activities. An ontology is also developed to include concepts and properties which may be applicable in the domain of surveillance and its applications. The proposed approach is validated with actual implementation and is thus evaluated by recognizing suspicious activity in an open parking space. The suspicious activity detection is formalized through inference rules and SPARQL queries. Eventually, Semantic Web Technology has proven to be a remarkable toolchain to interpret videos, thus opening novel possibilities for video scene representation, and detection of complex events, without any human involvement. The proposed novel approach can thus have representation of frame-level information of a video in structured representation and perform event detection while reducing storage and enhancing semantically-aided retrieval of video data.

Keywords: Smart city, data integration, data modeling, surveillance video, ontology, video semantics, video dataset, object tracking

DOI: 10.3233/SW-200393

Journal: Semantic Web, vol. 12, no. 3, pp. 467-491, 2021