Explaining rifle shooting factors through multi-sensor body tracking¹

1.1Aim and scope

There is a research gap in machine learning approaches for skeleton data that explain their reasoning, as well as in using machine learning for decision support within the sports shooting domain. This paper presents a fully encompassing workflow ranging from data collection in live shooting tasks, to sensor merging, data processing, and the construction of explainable machine learning models. The attention mechanism in the deep learning transformer architecture is used to gain insights into what the model focuses on when it makes certain predictions [22]. The study will attempt to answer the research question:

Can relevant factors for rifle shooting tasks be determined through posture and body movements?

It is expected that, as a result of the proposed work, an increased understanding is gained of how to model experiments for shooting scenarios that use multiple body tracking sensors. The study will identify the possibilities and limitations of the gathered skeleton data, as well as give an initial insight into how skeleton data from live shooting scenarios can be modelled to generate relevant explainable factors.

2.1Postural stability and shooting performance

Several studies have used force plate sensors to measure posture control in shooting tasks. However, these force plates are prohibitively expensive and can be impractical in situations outside controlled research studies and clinical environments [23]. The studies that have examined body tracking in shooting tasks have had a delimited approach, mainly focusing on correlating shooting performance to mean sway velocity calculated from joint angles [5]. Skilled and experienced shooters have generally shown a better postural stability than novices [2, 4, 3]. Gun barrel stability has also been correlated to postural stability by several studies [5, 4, 3]. Although some studies have correlated posture to shooting performance between individuals [4], other studies have only shown intra-individual correlations [6]. These conflicting results, as well as a general lack of studies examining the effects of the posture and movements of the entire body on shooting performance, motivate further study.

2.2Kinect and body tracking in scientific research

The Azure Kinect sensor [10] is a viable low-cost alternative to classical marker-based motion capture systems [19] as well as force plates [18] for measuring balance. Compared to motion capture suits, the Kinect has performed moderately to excellently depending on the tasks. The upside is that the participants are not restricted by wearing a motion capture suit. However, poorer tracking of the feet and ankles has been observed [16, 19]. The Kinect sensor holds up well as a balance measurement when compared to force plate sensors [18, 19], sometimes even outperforming them, with the upside of not limiting subjects to a stationary position. Downsides are limitations in accurately representing anterior-posterior swaying movements, overestimating large swaying movements, and having a framerate that is limiting for certain tasks [24].

Despite an exhaustive literature search, very few studies were found that use skeleton data to assess skill levels in humans. Most such studies have focused on isolated aspects of the body, such as angles between key joints in handball throws [20], or the take-off velocity of jumping motions [24].

Because of occluded joints forcing the Kinect sensor to estimate positions, Gao et al. [25] found that using two Kinect sensors placed in front of the target at an angle led to higher body tracking accuracy. Núñez et al. [26] proposed data augmentations on the skeleton data by shifting the graph slightly, because limited amounts of data would cause generalisability issues for machine learning models. Vemulapalli et al. [27] suggested that model performance could be improved by making the skeletons invariant to the absolute locations of subjects. They did this by transforming the coordinates to a body-centric view with the hip center as the origin, and the x-axis running parallel to the hip.

2.3Action classification

Action classification tasks use time-series skeleton data, often with deep learning techniques, to classify human actions such as jumping, running, throwing etc. This section provides a brief chronological survey of the area.

Du et al. [11] used a hierarchically bi-directional RNN approach, and found that it was beneficial to divide the skeleton into five body parts (arms, legs, torso) before fusing them iteratively into one feature representation. Liu et al. [28] achieved good results with Long Short Term Memory (LSTM) and hierarchical tree traversal techniques, in large part due to the LSTM’s inherent strengths in discarding irrelevant information such as weak sensor data. Adding the attention mechanism to this approach improved the LSTM’s global context memory and increased predictive performance [29]. Attention has also been able to locate the most discriminative time frames across longer sequences of skeletons, e.g. being able to identify the point at which the arm begins to stretch in a punching motion [30].

Ke et al. [12] used convolutional neural networks (CNNs) to classify actions with skeleton data, arguing that they could remember long time sequences better than LSTMs. They restructured skeleton sequences into three cylindrical coordinate channels, each composed of four frames. Convolutions on graphs have also been used to automatically extract feature maps from joints connected spatially between each other, as well as temporally through time [15]. Plizzari et al. [31] combined a high level local feature extracting Graph Convolutional Network (GCN) with transformers and self-attention, allowing for explanations of what features were mutually important for predictions, as well as alleviating the long temporal relation issues of LSTMs [22]. Similarly to Song et al. [30], they separated the temporal and spatial skeleton graphs into separate streams, which allowed the model to attend both to the important time frames and the joints with the most discriminative power. Transformers are composed solely of attention layers, and have an added benefit compared to other models of being able to explain their reasoning using the attention matrices produced from each attention layer [22].

3.1Experiment

The research question will be answered with the help of an experiment which will rely on data gathered from several human participants. The participants will perform a rifle shooting task, as described in the data collection section. The experiment is a quasi-experiment, meaning that there will be no random assignment, and that the potentially discovered relations in the data may not be true cause and effect relations [32]. The experiment will be controlled in the sense that there will be a full control over the manipulation of the data, and as many iterations as needed can be performed. The dependent variables will be the various shooting poses of participants as well as which orientation (strong or weak hand) the participants use. The independent variables will be the sequential body tracking data produced by the Azure Kinect sensors.

Figure 1.

Scenario illustration. Sequential illustration of the shooting scenario for the data collection. Source: [33].

3.2Data collection

The data collection for this study relied on one well-defined scenario, with a focus on identifying the postural effects on shooting performance with a semi-automatic rifle. The data collection was performed in an indoor shooting range to limit the effects of weather and wind on the participants and sensors. The scenario was designed to measure shooting performance when switching between different shooting targets and body postures (Fig. 1).

Scenario description:

Figure 3b shows the shooting target that was used in the data collection scenario. The participants were instructed to hit as close to the centre as possible while still maintaining a high speed of execution.

In order to build a robust and diverse dataset it would have been ideal to have shooting participants of different experience levels. However, due to safety constraints only experienced shooters with official weapons training could participate. 13 shooters were used in the data collection, 12 of whom had a military background, three of whom had a sports shooting background, and three of whom had a background in hunting. All participants were male, and ranged from the age of 31 to 62, with an average age of 48. In an attempt to simulate poor posture, the participants performed half of the scenarios with their weak hand. Each participant performed the scenario six times, resulting in a total number of 78 recorded shooting scenarios. Figure 4a,b shows a participant performing the scenario.

Figure 4.

Data collection participant and resulting skeletons. (a,b): A participant performing the data collection scenario, first standing (a) and then kneeling (b). (c,d): The resulting skeleton representations.

The Azure Kinect body tracking sensor [10] was used to capture the body movements of the participants during the scenario. The accuracy of the Kinect sensor can be affected by several factors such as occlusion of body parts, e.g. a hand behind the body from the view of the camera. Other factors include poor lighting conditions, or disruption of the sensors’ infrared signals from e.g. sunlight or infrared heaters. To mitigate the effects of poor sensor readings, three Kinect sensors were used: one behind the participant, and two to the front on either side of the participant at roughly a 45 degree angle, as suggested by Gao et al. [25]. The sensor behind the participant was placed slightly to the right, because when placed straight behind the participant the sensor would have difficulties with determining in which direction the participant was facing. To ensure a consistent and coordinated trigger timing of 30 frames per second on all three devices, the Kinect sensors were connected together with a synchronisation wire signalling when to capture new frames.

The Location of Miss and Hit (LOMAH) system (Fig. 3a) was used to record the time and position of shots by detecting the sound waves produced by bullets [34]. These shot positions were recorded in an external system as horizontal and vertical distance in millimetres from a defined centre of target position. A microphone worn on a backpack equipped on the participant detected shots and ensured that the Kinect data could be matched to the shots detected by the LOMAH system.

The large computational resource requirements of the Kinect sensors necessitated the use of two different machines to process the data, in order to retain the highest possible frame rate (30 hertz). This meant that the Kinect sensors were unaware of each other’s time systems, making it difficult to match which frames represented the same movements. Consequently, each participant was instructed to perform a synchronisation movement before each scenario by extending their non-shooting arm upwards and moving it slowly outwards in an arc away from the body and down to the side (Fig. 5).

Figure 5.

Skeleton synchronisation and merging. (a–c): A participant performing the synchronisation movement from the views of the front left (a), front right (b), and back (c) sensors, with the origins of their coordinate systems at the sensor positions. (d): A captured frame of the participant from the view of the front left sensor. (e): The merged version of the sensors, with the coordinate system aligned with the body positions.

3.3Data pre-processing

The Kinect produced 32 skeleton joints for each person present in each frame (roughly 30 frames per second). The joints broadly represented physical joints in a human body, e.g. hips, shoulders, knees etc. Because hand joints received poor estimations when further than 1.5 metres away from the sensor, they were excluded from the data. For each joint the following variables were collected:

Because the raw data from the three Kinect sensors were expressed in their own absolute cartesian coordinate system with the origin located at each sensor’s position (Fig. 5a-c), some pre-processing was required. The first steps involved matching the raw data from the three devices both temporally and spatially, and to isolate the body of interest, i.e., the shooter. A body-centric coordinate system was used, as suggested by Vemulapalli et al. [27]. The synchronisation movement performed by the participant was used to synchronise sensor data temporally across Kinect devices, and as a frame of reference for calculating new unit vectors for each device. The synchronisation skeleton frame index (S) and the following ten time frames (roughly 0.33 seconds) were used to calculate the new unit vectors, from which transformation matrices were constructed for each sensor device’s skeleton data. The normalised unit vector u^ (representing the new X axis) was defined as the average vector position between the left hip (⁢hl) and right hip (⁢hr) joints during the synchronisation sequence Eqs (1) and (2).

The normalised unit vector v^ (representing the new Y axis) was defined as the average position of the vector formed from the spine-chest (⁢gc) joint and the pelvis (⁢gp) joint during the synchronisation sequence Eqs (3) and (4).

The unit vector w^ (representing the new Z axis) was defined as the normalised cross product of the new X and Y axes Eqs (5) and (6).

The origin ⁢O used to translate coordinates to the new system was defined as the average pelvis (⁢gp) joint position during the synchronisation sequence (Eq. (7)).

Each joint vector ⁢ai from the entire skeleton sequence was then transformed into ⁢bi by computing the dot product of the translated joint position and the change-of-basis matrix. The change-of-basis matrix was constructed by appending the columns of the three unit vectors and taking the inverse of the resulting matrix Eq. (8).

Despite the sensor data being transformed into the same coordinate system, the joint estimations and body rotations from the three sensors differed slightly, which made skeleton merging difficult. To overcome this, the sensor on the side of the weapon-holding hand was used as a reference to attach body-centric joint positions from the other sensors to this sensor’s skeleton. These joint positions were calculated by performing new body-centric transformations on each skeleton frame for each sensor, ensuring that the body orientation was not affected by the differing sensor estimations. All joints with a high confidence value (i.e. not occluded by other body parts from the view of a sensor) were used to compute an average joint position ⁢ci. These average positions were then transformed and translated into ⁢di, to be represented in the coordinate system of the sensor on the side of the weapon-holding hand, using this sensor’s origin ⁢O and change-of-basis matrix [u^, v^, w^] Eq. (9).

Although three sensors were used in order to prevent the occlusion of body parts, joints were sometimes out of view from any of the sensors, resulting in outlier positions. To simulate the actual trajectory of these joints, their positions were estimated with a loosely fit 4th degree polynomial regression from the surrounding high confidence frames. To remove twitching joint movements, a median smoothing was performed on each frame from the surrounding five frames, followed by a mean smoothing from the surrounding three frames, resulting in a smooth merged skeleton representation (Fig. 5e). The shots detected by the LOMAH system were matched temporally to the skeleton data by detecting the shots with a microphone. Any samples containing shots that the microphone sensor failed to detect were discarded.

Features were extracted from the merged skeletons based on the angles of bones formed by the vectors between connected joints (e.g. the right femur bone was defined as the vector between the right hip and knee). This representation retained most of the information in the data, but made it invariant to the absolute position of the body in a coordinate system, allowing the model training on the data to focus on finding relevant patterns. The angle values for a bone were represented in three radian values defined by the angle value of the bone against the X, Y, and Z basis vector respectively (Fig. 6).

Figure 6.

Skeleton bone feature representation. Each bone was defined by three features per frame (X, Y, and Z angles), here illustrated for the right femur bone from a frame in a skeleton sequence.

Figure 7.

Model architecture. The input patches produced from a skeleton sequence, and how they were processed by the ViT model. Adapted from Dosovitskiy et al. [35].

To make the models more robust, a set of data augmentations were used: For each skeleton sequence, six new orientations were produced from rotations around the Y axis with 10, 20, and 40 degrees of rotation to both the left and the right, and used to train the model in addition to the original orientation, as suggested by Núñez et al. [26]. Gaussian noise was also added randomly during training to some of the bone features in an attempt to make the models more robust to imperfect data.

3.4Feature embedding and ViT adaptation

Because of the strength of Transformers on sequential data tasks, as well as their inherent explainability capabilities [22], a Vision Transformer (ViT) was chosen for the learning tasks. ViTs divide images into equally sized rectangular patches, and embed them into projected one-dimensional vector representations by feeding them through trainable embedding layers. These embeddings are then treated in the same way as positional tokens in standard transformers. This means that ViTs are not limited solely to use on images, but can be used on any data that can be represented with patch embeddings. ViTs can explain their predictions by using the attention produced from the patch embeddings [35].

Each skeleton sequence was represented with time frames as columns, and the 25 different bones as rows with the X, Y, and Z radian values stacked on each other in a fixed feature representation of 3 × 25 × 60 scalar values, i.e. a skeleton sequence of two seconds with 30 frames per second, and embedded this into patches of size 3 × 1 × 10. This meant that each patch represented one bone and its radian X, Y, and Z components over a time period of roughly one third of a second (Fig. 7). The feature representation was partly inspired by Ke et al. [12], who divided their data into time clips consisting of separated cylindrical coordinates. Because the patches were one row in height, representing only one bone, each bone for a moment of time could attend to any bone and time (including itself) across the patch embeddings. This allowed for the extraction of the attention for each bone and sub-sequence for a prediction, explaining some of the model’s reasoning. The ViT model used four multi-head attention layers composed of eight heads for each layer. A multilayer perceptron (MLP) head attached at the end of the model produced the output(s) for the specific task (Fig. 7). Each attention layer used a Gaussian Error Linear Unit (GELU) activation function. For both learning tasks, a dropout of 0.3 was used, as well as a reduce-on-plateau learning rate starting on 1 × 10-3, and reducing by a factor of 0.3 after plateauing for three epochs.

For each prediction, the model produced 32 attention matrices from the eight heads of the four attention layers, which were summed up into one representation of the total attention. Figure 8 shows how the attention map was processed from the raw attention map produced by the model into a format that was deemed easier for humans to interpret, explaining which joints affected the prediction for each time frame.

Figure 8.

Attention extraction. Pipeline describing the process from an input skeleton sequence to an explainable attention visualisation for a model prediction.

3.5Shooting pose estimation

A pose classification task was used for demonstrating the capabilities of the workflow presented in this study. Four pose classes were identified and labelled from the data collection scenario: standing shooting ready, standing aiming, switching position, and kneeling shooting. Estimating the pose of a shooter in a shooting scenario could be useful in a live prediction situation in a larger system. Such a system could potentially be made up of many different models trained for judging different shooting poses, and would therefore need some way of determining the current shooting pose. The labels were automatically assigned based on the timing of shots. The frames from 0.5 seconds before the first shot of a 3-shot sequence to the last shot of that sequence were labelled as standing aiming or kneeling aiming. Each 60 frame (two second) long skeleton sequence belonging entirely to one class label was extracted from the scenario. New sequences were extracted starting at every third frame. For this learning task, a cross entropy loss over four output nodes for the four classes was used, as well as a batch size of 256 samples. Because the classes were not equally distributed (Table 1), an imbalanced class data sampler was used to semi-randomly upsample the classes with few samples, and downsample the classes with many samples during training. The 13 participants were separated into 13 folds and used in a non-random repeated cross-validation. To obtain robust result, we trained 12 different models for each participant, where the samples from one other participant acted as the validation set one time each, and the remaining 11 participants acted as the training set. This resulted in a total of 156 models being trained for the experiment. We kept participants entirely in either test, validation or training to ensure that the model did not overfit to one specific participant during training. The training participant samples were fully augmented as described in the Section 3.3.

3.6Shooting hand and shooting performance estimation

Due to the relatively low amount of participants and data samples, we could not predict skill through continuously increasing skill levels in a regression task with the shooting score as the target variable. Instead, a binary classification task was constructed to test whether our approach could separate skilled and unskilled shooters from each other. The samples produced from the scenarios where the participants used their strong shooting hand were used to denote good posture, and the samples that used the weak hand were used to denote poor posture. The reasoning was that each shooter was much more accustomed to shooting with their strong hand, and would therefore make mistakes when shooting with their weak hand. The shooting accuracy and speed differences between strong and weak hand shooting showed that this was the case for most of our participants; with the exception of one participant the performance became worse when using the weak hand. The average time taken for a three shot series with the strong hand was 2.6 seconds with an average shot score of 2.19, whereas it took 3.4 seconds with an average shot score of 1.89 with the weak hand. Those who had a more similar performance between their strong and weak hand were also observed to have a more similar posture than those who had a large performance difference when using different shooting hands.

The same feature representation, model architecture, and non-random repeated cross-validation were used as in the pose estimation task (13 folds, 156 models). To limit the scope of this task, only standing shots were investigated. We extracted two-second samples by looking at the frames from 1.5 seconds before the first shot of a standing three shot series to 1.5 seconds after the last shot in the same series, extracting new sequences starting at every third frame. The class distribution was very close to equal between the two classes (Table 2). All skeletons were adjusted to a right-handed orientation by mirroring the left-handed shooting scenarios against the plane formed by the Y and Z axes, and switching the positions of the joints on the left and right side of the body. This ensured that the model could not learn to simply classify left- and right-handed scenarios, but that it would have to learn relevant factors of the actual poor posture that was produced from the shooters using their weak hand.

4.1Pose estimation

Table 3 shows the total results for the pose estimation task in the form of total results and the 95% confidence intervals based on the results from each fold (participant) in the experiment. Because the class distribution was imbalanced, both Accuracy and Cohen’s κ were used as evaluation metrics, as Accuracy would be biased towards majority classes, whereas Cohen’s κ measures how much better the prediction was than a random prediction given the class distributions [36]. It is worth noting that the confidence intervals were computed with unequal fold sizes; the participant with the most samples had 1792 samples, whereas the one with the fewest samples had only 815 samples. The model performed best on standing shooting ready and kneeling aiming, but often confused switching position with kneeling aiming, as well as standing aiming with standing shooting ready to some extent. These were reasonable errors, as these were the classes most similar to each other. In addition to this, the automatic labelling was not always 100% correct.

Attention statistics were calculated for the model by ranking the importance of each bone feature for each fold (participant), and then computing the average of these rankings across all participants. The attention often focused on the position of the arms or the femur bones (Table 4), which was reasonable, as the angle of the arms could help determine whether a participant was aiming or not, and the angles of the femur bones could help the model determine whether a participant was standing or kneeling. Figure 9 (switching position) shows how the processed attention produced from the model was visualised on a 3D skeleton model. This figure highlights how the model attended to the left femur bone in the visualised time frame, but also earlier in the sequence when the participant was in a standing position. This indicates that the model has learned to attend to the changing angle of the femur bones to determine whether the sequence involved switching from a standing to a kneeling position or not.

Figure 9.

Attention visualisation: Switching position. The input skeleton sequence of one of the participant switching from a standing to a kneeling position, the joint attention produced from the attention matrix from the pose estimation model, and a skeleton visualisation of the joint attentions for one time frame. Darker colour indicates stronger attention.

4.2Shooting hand and shooting performance estimation

Table 5 shows the results from the shooting hand estimation experiment. Overall, the results indicate that it is possible to some extent to separate participants using their strong hand from participants using their weak hand. However, a large difference was observed in prediction performance between the participants, as well as for each individual shooter depending on which other participant acted as validation data and thus stopping criterion for the training. It is not entirely clear why there was such a big difference in model performance between participants, but most likely more training data is needed. Another reason for the difference in performance could be the differing shooting styles, and too many pose faults in relation to the number of collected samples. The model may have learned simple indicators of good versus bad posture, but these could be difficult to generalise between shooters with e.g. aggressive or more relaxed poses.

Table 6 shows the computed attention statistics from all trained models for the shooting hand estimation task. These statistics indicate that the model has learned that the main difference in posture for strong versus weak hand shooting lies in the position of the forearm of the trigger arm. Other important features came from the head and centre torso body parts. This is in line with observations that were made from studying skeleton samples, where the shooting samples using the weak hand often had a more unnatural pose where the participants leaned into the weapon differently. Figure 10 highlights the difference in pose for one participant while actively firing the weapon during the scenario. This figure shows that the elbow of the trigger arm is raised when using the strong hand, whereas the participant is more huddled together and leaning unnaturally into the weapon with the weak hand. This is in line with many of the other top attended features from Table 6. Overall, domain experts noted an unrelaxed shooting position of the head and shoulders when studying individual samples using the weak shooting hand.

Figure 10.

Strong and weak hand comparison. Shows a comparison of one of the participants in the act of firing their weapon with their (a) strong hand and (b) weak hand. Highlighted are the top 10 bones from the overall attention statistics for the shooting hand task (Table 6). Note that all left-handed samples were swapped to a right-handed orientation.

An attempt was also made to predict skill by modelling a regression task where the target variable was the shooting score achieved by each participant, but this yielded close to random predictions. Most likely a lot more data is needed for such a task.

5.1Future work

There are many possibilities for future studies to expand on the work in this study. For better shooting performance estimation, there may be a need for more manual features based on expert knowledge of the shooting domain, such as mean sway velocity and other established factors [2, 4, 3]. Although deep learning models can generally find important features from raw data, the relatively small dataset could benefit from handcrafted features with more discriminative power. Such features could be used both in classical machine learning models or as additional dimensions in the architecture proposed in this study. Another future step could be to examine different ways of modelling shooting skill through classification or regression tasks. Different input sequence lengths would allow for studying both long- and short-term movement patterns. This could be done by warping inputs of different lengths to one size and including some time indication token, or through using multiple models with different input sizes, or models capable of dealing with varying input sizes. It could also be interesting to examine whether a different coordinate representation (e.g. cylindrical [12] or spherical) could have an effect on model performance.

If shooting performance estimation was more successful in the future, the attention explanations could be used as a direct visual feedback to a novice shooter in a product used in live training scenarios by highlighting important aspects of their posture control. This study has examined and demonstrated one of the ways in which attention maps can be visualised through simple sums of the attention placed on a bone in time, but there are many additional possibilities for explaining predictions with visualisations. Because it is known which pair of time/bone patch attend to each other (or themselves), visualisations of the attention between pairs of bones/joints could extend the explanations further. Attention could also help during model development as a tool to identify incorrectly learned patterns, or where the features were lacking. It would also be interesting to perform user studies with shooting instructors to determine whether the attention visualisations were helpful and intuitive, and use the expert assessment to improve explanations further.

Additional data quality improvement techniques such as smoothing could be examined further, as the poor joint estimations produced by the sensors were potentially detrimental to model performance. Other body tracking approaches such as body tracking suits could potentially be an alternative in order to produce more accurate skeleton representations [16, 19]. More data samples overall are probably needed, with increased diversity among shooter styles and skill levels. Different shooting scenarios with a bigger range of movements could also be of interest for future data collections.