A hybrid deep learning based approach for the prediction of social distancing among individuals in public places during Covid19 pandemic

1Introduction

The concept, ‘Social distancing’ has gained greater importance in the present day. Ever since the outbreak of the Covid19 pandemic, social distancing has been considered one of the most vital measures to break the chain of the spread of the virus. With the first and second waves of Covid19 hitting the globe and affecting nearly 180 countries across the globe, governments and health fraternities have realized that social distancing is the only way to prevent the spread of the disease and to break the chain of infections till a majority of the population are vaccinated. In the field of artificial intelligence, researchers are coming up with solutions that could be implemented in order to ensure that the Covid19 protocols are strictly followed by the people. However, there are cases in countries like India, France, Russia, and Italy where either people are heavily populated or do not adopt preventive measures like social distancing in crowded places [1]. Social distancing is a healthy practice or preventive technique which evidently provides protection from transmission of the Corona (Covid19) virus. It has been identified that when there is a maintenance of a minimum of 6 feet distance between two/more people [18] the spread of the virus could be avoided. Social distance doesn’t have to be always people’s preventive technique rather it could also be a practice that could be adhered to reduce physical contact towards transmitting diseases from a virus-affected person to another healthy being [9]. Deep learning like its role in every field has been found to create a significant impact in addressing this problem [4]. Manually monitoring, managing and maintaining distances between people in a crowded environment like colleges, schools, shopping marts and malls, universities, airports, hospitals and healthcare centres, parks, restaurants and more places would be impractical and therefore adopting the machine language, AI and deep learning techniques in automatic detection for social-distancing is essential. Object detection with RCNN in machine language-based models had been adopted by researchers since it offers the investigators faster detecting options. Faster RCNN is found to be more advantageous, faster convergence and also provides higher performance [3] even under a low light environment.

In the regions of America, Europe and South-East Asia due to poor social distancing and improper measures against Covid19 by the people, in the years 2020 and 2021 many fatalities were recorded and most of them which were because of the violation of the threshold in social distancing measures during Covid19 [10]. The concept of adherence to social distancing has been implemented by machine learning researchers in the past through techniques like face recognition, object (human) identification, people monitoring, object identification, image processing and so on [29, 32].

Deep learning and machine learning-based based Artificial Intelligence (AI) models surpass various criteria like time consumption, cost reduction and reduction of human intervention. Machine learning-based solutions help in achieving results with greater accuracy, minimal loss and errors in addition to reducing the prediction time from several years to a few days [16]. Majorly R-CNN (region-based CNN) and faster R-CNN are applied for accurate and faster predictions. To measure the predicted outcomes, researchers generally utilize the metric evaluation approaches/ techniques like regressors (Random-Forest-Regressor, Linear Regressor), IOU (intersection-over-union) and more. For object detection, the IOU approach is commonly adopted to evaluate performance.

The uses and scope of RCNN in object detection models are mainly to identify/detect the image/video and bounds the class targeted by researchers. The bounding box separates the class needed from other classes and objects in an image. Thus to validate the bounding box, investigators can code them with colors (for instance green for no error, red for error) and also use RCNN to label the images under categorized files [7]. Similarly, the RCNN application is found to be majorly identified in the defense system by the militaries, automobiles using object detection like Tesla, surveillance cameras and securities in banking, malls, hospitals, jewelry shops, and more that are crowded and need an advanced security system [27]. Recently, due to advancements in technology and the advantages of surveillance, shops with larger-area to monitor have adopted surveillance cameras (with heat, objects, light, and other sensory detections) to secure their shops [11]. Thus RCNN in object detection has higher scope in identifying, classifying and labeling images/videos.

This research has developed a customized deep learning RCNN-based architecture. A hybrid approach combining Detectron2 and Intersection over Union method has been applied in order to predict whether the social distance is maintained by the individuals or not, especially in crowded places and public places.

Purpose: The study is mainly motivated towards examining and analyzing the social distancing among the individuals (i.e. person class), in public and crowded areas with the use of RCNN architecture where the IOU method is used for evaluating the performance of the developed model. Detectron2 has been used to identify ‘person class’ from other classes in an image to measure the social distance between the identified class, people. A similar research as a ‘case study’ approach would also be examined for adaptation of deep learning and object identification through image as input to measure social distancing between individuals.

2Related works and research background

Social distancing post Covid19 was vigorously studied by many researchers where prediction models were developed to maintain and monitor social distancing. Authors [31] focused on the prediction of social distancing post Covid19 outbreak in India with DNN architecture. [19] utilized the TWILIO and the authors [31] utilized the YOLOv3 as their base architectures respectively for addressing the same problem. These studies utilized the faster R-CNN and found that the architecture is more flexible and faster than other existing approaches. The conclusions from the studies revealed that efficient and effective object-detection outcome is attained with the YOLOv3 with faster R-CNN model. However authors [14] developed a YOLOv4-based ML model and tested the model with COCO datasets. Their model architecture was also based on DNN (Deep-Neural-network). The latter model was found to render 99.8% accuracy with a speed of 24.1 frames per second (fps) when tested on a real-time basis. Authors [26] had developed an AI-based social distancing model which measured the distances among two-or-more people with social-distancing norms. The authors adopted YOLOv2 with the Fast R-CNN for detecting the objects. The objects considered by them were ‘Humans’ through person class. The accuracy was 95.6%, precision was 95% and recall rate was 96%.

Authors [22] examined the different techniques of metrics in evaluating the prediction models to monitor the social distancing via ‘Computer-vision’ as an extensive-analysis with existing social distancing reviews. They concluded that mAP is found effective in evaluating the prediction models. The study also focused on examining the performances of security-threats identification and facial expressions-based models that had adopted deep learning and computer vision, through real-time datasets with video-streams. The authors found ‘YOLO’ to be effective in the detection models among other AI-based models. Henceforth, from the study it’s streamlined that, adopting two-stage object detectors is wiser, efficient, effective and accurate.

The study by [6] investigated the social-distancing concept, post Covid19, by developing the vision and critical-density based detection system. They developed the model based on two major criteria. First criterion was to identify violations in social-distancing via real-time vision-based monitoring and communicating the same through deep-learning-model. Second the precautionary measures were offered as audio-visual cues, through the model towards minimizing the violation threshold to 0.0, without manual supervision by reducing the threats and increasing the social distancing. The study adopted YOLOv4 and Faster R-CNN where the precision average (mAP) was achieved higher in BB-bottom method with 95.36%. Similarly, accuracy at 92.80% and recall score at 95.94% were also achieved.

The study [10, 11] focused on deep-learning architecture that utilizes the social-distancing concept as a base for their evaluation with people monitoring/ management as a secondary focus, post the Covid19. The authors [10] have utilized YOLOv3 for identifying humans and faster R-CNN as the social-distancing algorithm. The model achieved 92% accuracy in tracking, without transfer-learning and 98% with transfer-learning; similarly, 95% of tracking accuracy. The study insisted that social-distancing detection through YOLO with tracking transfer-learning technique is highly effective than other deep-learning models. Authors [17] proposed a study to monitor Covid19 based social distance with person detection and tracking with fine-tuned YOLOv3 and Deep sort techniques. The authors developed the deep learning model for task automation, to supervise social distances between people, using surveillance videos as datasets. The model used YOLOv3 for object detection, towards segregating background and foreground people; similarly, deep sort methods were utilized towards tracking the identified people, with assigned identities and bounding boxes. The deep sort tracking method together with YOLOv3 showed good outcomes with balanced scores of fps (frames-per-second) and mAP (mean-Average-Precision) towards supervising real-time social-distancing, among people.

Similarly a study by [3] proposed an advanced social-distance monitoring model to surpass the drawback of existing models where low light surroundings are the major hindrance in the identification of people, as the foreground. Identifying and monitoring people, especially in crowded and low-light environments in deep-learning model prediction models, especially in circumstances like Covid19 and other pandemic circumstances has made researchers to develop new models and improvise existing models, where social distancing is a must. The developing disease caused by the SARS-covid-2 virus (Covid19) has acquired a worldwide crisis, with its deadly distribution, all-over the globe. Due to isolation and social distancing, people try to walk out of their homes, during the nighttime with their families to breathe in some fresh air. In such circumstances, it is essential to consider efficient steps towards supervising the criteria of safety distancing to avoid positive cases and to manage the death toll.

Thus, from the existing researches and studies it is observed that YOLO for object detection has been adopted by the investigators majorly, with RCNN and Faster RCNN in their architecture. Though YOLO detects the objects meaningfully, it lacks behind in detection speed and thus affects the tracking process. Whereas, according to [24] the Detectron2 model is found effective and rapid in tracking processes than YOLO. Thus, in this research, a deep learning-based method is proposed using the RCNN with Detetcron2 for measuring social distances among individuals and detection of real-time object-class (i.e. person class) with a single motionless ToF (time of flight) camera.

The below Table 1 shows the reviews of social distance prediction during covid-19:

Through the reviews it is evidently understandable that, YOLO with CNN, R-CNN and Faster R-CNN are majorly utilized in the recent years, towards social-distancing based object detection models. However the usage of models with different neural networks, algorithm and evaluation metrics are observed to be lesser through the reviews undertaken.

Research gap: There have been several research works on solutions to overcome the impact of pandemic after the outbreak of Covid19, especially through social distancing as a norm. Numerous machine learning studies were attempted to predict the social distancing in public areas. Models in predicting the social distances by authors [21] focused on hospitals, ATMs and malls as crowded area with DNN model; [8] developed a stereo-vision based model for monitoring and predicting social distances; [28] developed a model that predicts social distancing and monitors face-masks in people with object detection based mobilenetv2 model; [20] developed an IoT with Passive-Infrared sensors based social distancing prediction model; [25] developed a computer-vision based prediction model; [2] developed a social-distancing prediction model with image depth estimation; [15] developed a Infrared-array based sensor model that detects and monitors social distancing; authors [33] developed a UAV (unmanned-aerial vehicle) based PeelNet incorporated social distancing model; [12] developed a VSD model through automated estimation of distances based on inter-personal spaces between the people identified; [30] developed a MobileNet based model that predicts social distancing through sound-wave technique and [13] developed a parallel deep learning model that used BP-PSO (Back-propagation-particle swarm optimization) and community detection techniques.

The present study has developed a novel approach to detect whether social distancing has been maintained by individuals at public places. The current approach has combined “Detectron2” and “IOU’ techniques and developed an R-CNN based customized machine learning model for detecting the social-distancing between people, especially in the crowded areas. This research has developing and tested a customized model for social distancing prediction, through video object detection using COCO dataset.

Novelty of the work: Authors [10] have proposed and implemented an approach for prediction of social distance. His model adapted YOLOv3 as the base model for transfer learning. The model rendered an accuracy of 95%. Similarly the authors [23] have proposed an approach for social distance prediction and mask prediction using IoT using Faster RCNN and IoT. However, they had not developed any novel model. Authors [19] proposed a model with R-CNN architecture where TWILIO as communication-platform. The model achieved 96% accuracy with loss of 3%. Contrarily, authors [6, 26] each developed R-CNN model with YOLOv4 and YOLOv2 respectively, in predicting social distancing between people in crowded areas, aiming at achieving higher accuracy than existing R-CNN based YOLO models. Howsoever the both studies achieved 95% accuracy, which was not higher than existing models.

In this research, the author has developed a Detectron2 model with IOU metrics for predicting the social distancing between individuals in crowded public areas that has not been attempted by any authors, yet. This paves way in the research to adopt new evaluation techniques and prediction models than attempting to adopt existing models and comparing with other similar models.

3Proposed system

The system proposed here is a social distancing technique for analyzing objects that utilizes the deep learning, computer vision along with python language, for detecting intervals (distance) among people for maintaining safety and monitoring crowd. RCNN was used to develop the model along with deep learning algorithms, computer vision and CNN (convolutional neural networks). To detect objects (people) from video, object detection method was used where ‘Detectron’ algorithm is applied. From CoCo datasets and obtained results with class-as-objects “People” as the class was alone filtered, by disregarding other classes. To map people from other classes in the video or frame, bounding boxes are mapped with different colors. Finally, distances between identified people are measured through obtained outcomes.

Proposed approach: The research focuses on social-distancing between people through video frames as dataset. The datasets are obtained from existing library and programming language used is python. Faster RCNN is used for developing the model with detectron2. Object detection algorithm is used to differentiate humans (people) from other classifications. The approach proposed is represented through flow-diagram in the following section.

Flow-diagram: The flow-diagram (refer to Fig. 1) represents the research approach in detailed manner:

Fig. 1

Flow-chart of proposed research.

Proposed Architecture: Among the R-CNN, faster R-CNN and fast R-CNN, faster R-CNN is viewed and identified as effective in terms of performance. Though there are numerous methods and approaches in object detection using machine language based deep learning, faster R-CNN with Detectron based algorithm has been proved to be effective than the previous models [18].

The architecture (refer to Fig. 2) is developed with faster R-CNN. The reason for adopting and utilizing faster R-CNN as the backbone in the developed model is for its features namely: accuracy, precision and speed. The developed model architecture includes 5 Convolutional layers with a configuration of 128, 256, 512 and 1024. The research includes deep learning with 8layers of ReLU activation function. 3*3Conv Kernel size has been adapted.

Fig. 2

Architecture of the customized detectron2 with RCNN model.

The image is fed as input in the first layer of proposed detectron model with image size of 224x224x3. In the middle layer of the model, 2d convolutional, batch normalization ReLU and max-pooling are included. The convolutional layer here maps image features, where the filter is set-as 5x5. The width along with height of image in regions, is defined through filters. In regularizing model and eradicating the overfitting challenge/issue, the batch normalization is included in the model as middle layer. Later, the ReLU loss-function is used in the neural-network to introduce non-linearity. Finally, the max-pooling were included to obtain down-sampled images in pooling-region.

The stride-size is set at 2x2. Detectron2 ResNet50 is utilized for object detection in the model with similar layers.

Based on the architecture the study adopts the following schematic representation (refer to Fig. 3). The image is passed through the convolutional layers (Backbone network) where RPN transforms the image as feature maps. In the transformation RPN (Regional proposal network) identifies the regions with objects known as ‘proposals’. Identified proposals will be examined for feature extraction, with RPN as backbone network. If there are no proposals found, the ROI extracts features directly from RPN and passes on to classifier for object detection network layer. Thus the faster-RCNN in this research is developed for object detection along with Detectron.

Fig. 3

RCNN schematic diagram.

Model developed in Classifying Social Distancing: In the developed model the following algorithms were used:

RCNN: The faster R-CNN for the social-distancing and object detection is carried through the following algorithm:

Step 1: Initially, cloning the repository for the Faster R-CNN implementation is carried out;

Step 2: The folders (training datasets and testing datasets) along with the training file (.csv) is loaded to the cloned-repository;

Step 3: Next,.CSV file is converted into.txt format/file with new set of data-frame and the model is then trained using train_frcnn.py as file in python keras;

Step 4: Finally, the outcomes i.e. the predicted images with detected objects as per the norms in the codes and the results are saved in a separate folder as text images with bounding box.

Detection: The algorithm for the developed social-distancing evaluation through object detection for detedctron2 with faster R-CNN50 and 101 is designed as:

Step 1: First, the images from COCO-datasets are loaded and accessed as inputs for the developed model;

Step 2: Initially the images are passed through the ConvNet of Faster R-CNNs of 50 and 101;

Step 3: Merge the model files from faster R-CNN-50 and train the sample-dataset and initialize the training;

Step 4: Choose the Learning-Rate with good outcome; here 300 is predicted to be enough for the obtained sample dataset;

Step 5: Next, focus on maintaining the learning rate by preventing the decaying of learning-rate;

Step 6: Set the ROI-head size as 512 for training datasets and label ‘person class’ is alone selected and the metric is obtained;

Step 7: Once the datasets are trained, the model is tested with remaining images and bounded through object detection;

Step 8: The same process is repeated in testing and if the identified person class satisfies the norms for social distancing criteria; where ‘green bounding-box’ is applied for the image with proper social-distancing and ‘red bounding-box’ is applied onto image with no social-distancing. Finally, the results obtained are stored in a file under the person-class folder.

Thus the social-distancing along with detecting people (person-class) upon video frames as images is done in this research. The steps involved are represented in the following illustration (refer to Fig. 4).

Fig. 4

Classification and social-distancing of person-class.

IOU: To predict and bound the class in identified object class (person) the IOU is used and the model’s predicted results are compared with estimated outcome. IOU is also used for metric evaluation to evaluate the model’s precision and accuracy. The IOU (Intersection-over-Union) is calculated through algorithm:

Step 1: Initialize epsilon at 0.5 (1e-5) where the two boxes are pre-defined as lists;

Step 2: Represent the bounding box, with IOU values with ‘X1,Y1’ as box’s upper-left corner and ‘Y2,X2’ as upper-right corner;

Step 3: Initially the value is set at 0 and after each-iteration the value is increased by 1;

Step 4: Later, estimate the area through overlapping of boxes where height (Y2-Y1) and width (X2-X1) intersects. If there are no overlapping (i.e. intersection of boxes) then the value is set at 0;

Step 5: Combined areas are then measured as two-sections, firstly, sum of a (a[2]-a[0]) x (a[3]-a[1]) and b (b[2]-b[0]) x (b[3]-b[1]), secondly, by estimating (area_a)+(area_b) - (area_overlap); similarly combined area-ratio is measured with IOU estimation (area_overlap) / (area_combined+epsilon);

Step 6: Boundingbox is estimated and applied on the input-image to classify the object (person-class) to estimate social-distancing;

Thus IOU is estimated through bounding-box intersection (refer to Fig. 5) as:

Fig. 5

IOU estimation in object detection.

Generally, the more the bounding-boxes overlaps the excellent the score is and vice-versa. Hence in the research, 0.75 as threshold for bounding-box in object detection for person class is used in IOU metric evaluation.

4Methodology

The research is done to measure the social distancing between people as “person class” through object detection method in deep learning technique. For the proposed aim, the developed model adopts neural networking (RCNN) and deep learning with python as programming-language. Initially the datasets are acquired, trained-and-tested and evaluated for its efficiency through IOU evaluation method. For detection of objects (‘person class’) ‘detectron2’ is utilized.

Datasets Used: The research uses the video streams and frames for detecting objects to identify the target-object (people as person-class) class from the other class-categories; henceforth, person class alone will be used for identifying and detecting objects for the developed model. The research is to examine the social distancing between people and thus other categories would be ignored.

The datasets were obtained from COCO dataset. The dataset has 95categories of classes (refer Table 2), where, researchers opt for required object detection classes alone.

Here ‘person’ as class from Humans category is alone opted and other 9categories are ignored for the prediction model. The data is obtained from open source through “Kaggle” [5].

Training: The model developed is trained with small sample datasets, to test the performance and accuracy of predictions made. If the outcomes are unsatisfactory the model is modified according to the Training:

Input: COCO 2017 dataset

Output: Bounding boxes around objects detected

Followed by the training phase, the research uses the SGD algorithm for gradient descent in developed model.

Overview of gradient descent (GD): The research utilizes the SGD algorithm for its implementation and efficient usage in neural networking. However, it lacks in two features. Firstly, it requires large iterative numbers and regularization parameters as hyper-parameters and secondly, its hyper-sensitive towards feature scaling.

Generally, the GD variants could be basically categorized as SGD (stochastic gradient descent (SGD), BGD (batch gradient descent) and MBGD (mini-batch gradient descent). When compared against other gradient descents (Eq.1 and 2), the SGD (Eq.3) is found effective as loss function for object detection. The BGD also known among researchers as ‘Vanilla GD’ is formulated as:

This formula is mostly utilized for cost function gradient by researchers since this doesn’t allows the researcher to update gradients more than once and thus slower in operation processes.

The MBGD is formulated as:

Though the MBGD has stable convergence it allows only smaller batches (50in size) for evaluation and thus SGD is mostly preferred by researchers. The SGD formula utilized here is:

Testing: The testing of model is carried-out through the following steps:

Input: Images from a surveillance camera

Output: Detecting social distancing amongst people

Loss Function and other formulae used:

a) ReLU: The ReLU (Rectified Linear Units) is better and performs higher than Tanh or Sigmoid in Loss Function estimation. Here, the ReLU is estimated since it rectifies and evades vanishing-gradient issues. Similarly, it is less-expensive when computations are compared against Tanh and Sigmoid and thus researchers adopt ReLU loss function a lot; it also provides simpler, easier and faster mathematical operations.

However, it has certain drawbacks, like: it shouldn’t be utilized out of hidden-neural layers of network model, fragile gradient loss during training with dead neurons, activation blow-ups and ReLU might stop replying to error variations when gradient range is 0 (equation 4).

The formula for ReLU activation could be measured through:

b) R-CNN: (Regional –Convolutional Neural Network): The research makes use of bounding-box regression to improvise the performance of localization. The ground-truth prediction of bounding-box localization is estimated with its size and location, through the formula:

where the initial two-equations (5a and b) specifies the translation of scale-invariant of P’s centre (a and b) and the latter two-equations (5c and d) specifies transformation of log space with the height (f) and width (v).

c) IOU (intersection-over-Union): The IOU is measured through dividing the area of bounding boxes, where: overlapping (intersection of bounding-boxes) is divided by union (combined area of bounding-boxes). The IOU is estimated through [34]:

5Results and performance evaluation

Random Sample Dataset: The datasets for training the model are selected randomly (refer to Fig. 6a, b and c) to predict its reliability and accuracy towards comparing the results obtained against estimated outcomes.

Fig. 6

Input dataset selection.

Ground truth versus predicted by model: The following outcomes (predictions) are the results obtained from the trained-tested-model, where the ground truth is compared and weighed against it’ predictions, based on the algorithm. The results from trained detectron2 with faster R-CNN model are as follows:

a) Classification of Individual Person: The figure (refer to Fig. 7a) represents the ground truth of an identified person in the image versus the Fig. 7b representing the prediction made by the model by identifying the person-class only. It could be inferred that obtained accuracy and precision are similar to the estimated outcomes.

Fig. 7

Individual as person-class identification.

Interpretation: However, the study is focused on social-distancing with green-bound-box as correct distance and red-bound-box as incorrect distance, between two-or-more people. Since there is no more-than single person in the image, the algorithm identified the individual with green-bound-box, stating that, norms of social-distancing are satisfied by the ‘person’.

b) Classification of Group of People versus Objects: The following figure (refer to Fig. 8a) represents the ground truth of identified person in the image versus Fig. 8b representing the prediction made by the model with identification of person-class only.

Fig. 8

Group as person-class identification.

Interpretation: The image is of mass-people, where the ground truth identified 8invidiual people in Fig. 8a with class-person. The predicted model provided outcomes (8b) with same head-counts of 8 people as ‘person-class’. The model also identified that there is no social-distancing between the people and thus the resulted image 8b is obtained with red-bounding-boxes.

c) Classification of People and Animals: The figure (refer to Fig. 9a) represents the ground truth of identified person in the picture versus the Fig. 9b representing the prediction made by the model with identifying person-class only.

Fig. 9

Identified person-class social distancing.

Interpretation: The class ‘person’ is identified from other class-categories and evaluated for social distancing. According to the developed algorithm, only people-class is identified for social-distancing evaluation in 9a; thus the predicted outcome is found to be negative, where there is no social-distancing between ‘identified persons’ in the result 9b.

d) Classification of people from other categories:

Interpretation: From the figures (refer to Fig. 10a, b, f and g), it is interpreted that, there are people who maintained social-distancing and thus the green-bound-box; contrarily the remaining Fig. 10c, d and e has red-bounding-box indicating that there are no social-distancing between the identified person-class.

Fig. 10

Prediction and social-distancing by developed model.

6Case study with respect to Textile shops

Case study based on the developed model: The following example illustrates how the developed model can be used to check and measure the social distancing between people on a real time basis. The model is trained and tested with images of the people in textile shop in order to illustrate whether social distance has been followed by people in a crowded textile shop. Few images as samples of the people in the textile shop were tested and trained for social distancing. The results are:

Interpretation: The developed model can be explained as below. The green-bound-box represent the proper distance among individuals (people) and the red-bound-box denotes there exists no proper distance between two-or-more people.

Since, there is just one person in the image (refer to Fig. 11-before), the model identified the person with green-bound-box (refer to Fig. 11-after), stating that the norms of social-distances between people is justified by the identified ‘person’.

Fig. 11

Identification of individuals (with person class).

Interpretation: The model identified the mass/people (refer to Fig. 12-before), where 8 individuals under person-class has been identified. The model predicted 8 head-counts of the people as ‘person-class’ and bounded them with ‘person’ label; according to the social distancing algorithm it resulted with 8 red-bounding boxes (refer to Fig. 12-after), where it indicates there is no social-distancing between them.

Fig. 12

Identification of people in crowded areas.

Interpretation: Figures 13(a–f) denotes that, there are only few people with proper social-distancing in the textile shops, thus the green-bounding-box. However, the red-bounding-box in the Fig. 13(a–f) represents that improper or no social-distancing among the individuals identified as person-class.

Fig. 13

Measuring the social-distancing through the developed model.

f) Loss function: Based on the estimated versus obtained outcomes, the graphical representation of loss function (the time, data time, total loss and eta-seconds) of the developed model had been evaluated in python and represented below in the graphical representation of the Fig. 14a–d:

Fig. 14

(a) Time, (b) Data time, (c) ETA seconds and (d) Total loss.

Interpretation: It is inferred through the figures (refer to Fig. 14a–d) that no huge variations have been identified between the results obtained from the model. Similarly, the estimated outcome and obtained result of total loss is less at 0.1, thus proving that developed model is effective in predictions.

The figures (refer to Fig. 15a and b) denote the anchor values (negative-positive), whereas the Fig. 15c and d represents the Background samples and Foreground samples of the input images, for training the model.

Fig. 15

RPN number (a) negative anchors and (b) positive anchors; ROI-head number (c) BG-samples and (d) FG samples.

Interpretation: From the above figures (refer to Fig. 15a–d) it’s inferred that there are no huge variations in the rpn (region-proposal-networks) numbers of negative and positive anchors; similarly, the roi_head-numbers (region-of-interest) for foreground and background samples in pooling have no huge-variations. The predicted outcomes and ground results are similar. Thus the developed model is a good-fit.

The Fig. 16a–c represents the loss_rpn with loc (loss-rpn-localization), class and regression of bounding box outcomes. The results evidently insist that the regions are overlapped. Thus NMS (non-maximum suppressions) is used towards minimizing the proposal numbers. The loss is minimized at 0.1, as total loss of the developed model.

Fig. 16

Loss RPN (a) LOC, (b) Class and (c) Box regression.

The outcomes from Fig. 17a–c denote that, faster R-CNN outcome of accuracy is achieved at 98%, where the foreground-accuracy is achieved at 10% and false-negative at 0.1%. Concluding that, the model is accurate and precise in detecting objects and measuring social-distancing. Obtained results were higher than estimated outcomes with 75% (threshold value) and above stating the performance was effective.

Fig. 17

Faster RCNN (a) False negative, (b) FG class accuracy and c) Class accuracy.

g) Scheduling LR: The learning rate (as shown in Fig. 18) is set-to 4k intervals where the developed model attained successful learning rate at 1000k. It remained the same till 23k iterations stating that, there are no sudden dropping-down in the learning rate, but rather, the LR steadily decreased and remained constant from 1000k-70k iterations.

Fig. 18

Learning rate.

Scores post testing and training the model: The scores for the developed model after testing and training the processed dataset have been obtained (Table 3) and through outcome values the study concludes that, the detectron2 model with faster RCNN as architecture where IOU metric is adopted to evaluate the model.

h) Performance metrics: The object detection (human) in the model developed has acquired recall rate of 87% and precision of 97.9% which is more than average 75% stating that the model is a success with effective precision with minimal total loss of 0.1 and mAP at 84.5%; where existing models (refer Table 4) lack precision in human detection towards social-distancing threshold violation measures (refer Fig. 19).

Fig. 19

Performance evaluation and comparison with existing approaches.

Inference: The Fig. 19 exemplifies that among the existing models the developed model with detectron2 with faster R-CNN architecture acquired higher precision rate of 97.90% (98%) than other models, where:

Thus the researcher examined and evaluated the datasets with developed detetcron2 model where it’s evidently concluded that, the developed model is a success and good-fit for object detection based analysis models and for violation threshold based applications in object detection and monitoring. Majorly for evaluating the social-distance criterion the model is reliable, accurate, precise and also has a fine recall score (87%) with better mAP (84.5%) that exceeds the average score of existing models.

7Conclusion and future recommendation

Conclusion: The investigation particularly aimed at analyzing and developing a better model with object detection-based machine language adopted architecture where it could attain a higher precision rate than average metric scores attained by the existing models. The study adopted an IOU metric evaluation that uses mAP (mean-average of precision) which examines and evaluates the developed object detection model. The model developed is of detectron2 and faster R-CNN, where outcomes are estimated through ground-truth detection and prediction with an algorithm focused on faster R-CNN for social-distancing norms and detetcron2 for identifying the human ‘person class’ towards estimating and evaluating the violation-threat criteria where the threshold (i.e. 0.75) is calculated. The model attained precision at 98% approximately (97.9%) with an 87% recall score where IOU was at 0.5.

Limitations: The research has exclusively focused on identifying whether social distance has been maintained between individuals in crowded areas. This study does not involve other applications like detection of face mask, hand hygiene and so on.

Future work: In the future, the developed model could be converted into an android application and integrated with IoT-based hardware such that images of people in public places are captured on a dynamic basis and proper notification is given to the relevant authorities when an individual fails to adapt social distance at a crowded place.