Vehicle side-slip angle estimation under snowy conditions using machine learning

2.1Vehicle side-slip angle estimation problem

Centered on a vehicle’s mass, its motion is characterized in a horizontal celestial system. The overall velocity vc, at the vehicle’s center of mass, is a composite of longitudinal vx and lateral vy velocities. The angular velocity around the yaw direction z-axis is defined as yaw-rate Ψ˙, while Φ˙ and Θ˙ depict rotational speeds in roll and pitch directions. Side-slip angle β, a pivotal metric in vehicle dynamics, describes the deviation between the vehicle’s longitudinal axis and its actual direction of movement, specifically representing the angle between v and the vehicle’s orientation. If the vehicle drifts along the velocity vector at a 90-degree angle, β indicates a perpendicular turn of the vehicle’s longitudinal axis to the left.

β can be calculated using trigonometry if we could determine the center of mass and the vector components (vx, vy) of the total velocity vC at the vehicle’s center of mass accurately, as shown in Eq. (1).

with

However, accurate determinations are impracticable, which necessitates the exploration of alternative VSA estimation methods. Typically, VSA ranges from -4 to 4 degrees during regular driving [8]. Dynamic factors such as tire forces, roll angle, and kinematic aspects like vehicle mass or structure significantly influence VSA estimation. Observer-based methods often utilize these values, but their limitations are apparent due to issues with measurement accuracy and road-surface condition detection. Machine learning offers promising solutions for these challenges and relevant literature will be discussed in the next section.

2.2Machine learning essentials: CNNs and GRUs

Machine learning models, particularly CNNs and GRUs, have become increasingly popular in the realm of artificial intelligence in recent years. While CNNs have proven their proficiency in tasks such as human action recognition, object detection, and natural language processing, GRUs have found applications in time series forecasting [24], semantic analysis [25], and also natural language processing [26]. In the realm of VSA estimation, machine learning models, especially CNNs and GRUs, offer significant advantages in contrast to traditional approaches. CNNs aim to extract spatial features from images which makes them suitable for processing visual data from cameras to detect road conditions and vehicle dynamics. On the other hand, GRUs are adept at handling sequential data capturing the temporal dependencies in VSA estimations over time.

2.2.2Gated recurrent unit

Figure 2.

Single GRU cell architecture. This showcases the primary components including the input layer that receives data, the hidden state carrying information through the network, the reset gate which determines the extent to which previous information is forgotten, and the update gate that decides how much of the current state should be updated with the new proposed state.

GRUs, introduced in [13], are a type of RNN [12] designed to capture long-term dependencies in sequential data while overcoming the short-term memory limitations of traditional RNNs. They have comparable accuracy to other RNN structures like LSTM [14], but offer faster training and prediction due to fewer model parameters [29, 30].

Figure 3.

Comparison between the Ackermann steering model and single-track model.

As Fig. 2 illustrates, the GRU cell is governed by:

(2)

rt=𝑠𝑖𝑔𝑚⁢(Wx⁢r⁢xt+Wh⁢r⁢ht-1+br)

(3)

zt=𝑠𝑖𝑔𝑚⁢(Wx⁢z⁢xt+Wh⁢z⁢ht-1+bz)

(4)

ht~=𝑡𝑎𝑛ℎ⁢(Wx⁢h⁢xt+Wh⁢h⁢(rt⊙ht-1)+bh)

(5)

ht=zt⊙ht-1+(1-zt)⊙ht~

where xt, rt, zt, ht~, ht are the input vector, reset gate, update gate, candidate hidden state vector, sigm is the sigmoid function, and output vector at time t, respectively.

2.3Single track model

The single-track model, a physics-based kinematic model often used as a simplified version of the Ackermann steering model [23, 32], delivers a plausible representation of vehicle behavior without requiring extensive modeling or parameterization [22, 32]. This kinematic model was proposed for lateral accelerations below 0.5 g [32].

The bicycle model, a simplified version of the Ackermann model, as depicted in Fig. 3a, merges the front and rear wheels into single points each. This enables the model to depict lateral vehicle dynamics in a physically plausible manner as illustrated in Fig. 3b [33, 22]. The single-track model’s acceleration around the center of gravity is given by:

In the single-track model, the vehicle’s mass is assumed to be concentrated at its center of gravity, a common simplification in vehicle dynamics. While this assumption streamlines the model, it may overlook some real-world complexities. However, neural networks, especially CNNs and GRUs, can learn and account for such intricacies. Leveraging the deep learning capabilities highlighted by [34], our hybrid CNN-GRU architecture learns transformations, including those related to the center of mass, enhancing the model’s adaptability and accuracy.

4.1Experimental setup and simulation environment

We used the Unity3D based 3DCoAutoSim [42] platform to simulate the environment with snowfall, friction, and car tracks on the snow. The test ground setup used in this work is based on [10], which is a snow covered 4-lane dual carriageway circular test track. The friction between the road surface and the wheels is directly achieved by setting the Unity WheelColliders, which will be explained in more detail later. The setting of the test ground is mainly to extract visual data such as car tracks on the snow. For the dataset creation we performed multiple laps, in both directions of the course, and stored the sensor data in separate rosbag files.

The vehicle model in the simulator corresponded to a Toyota Rav4 Hybrid (2020) [43] and closely resembles it in terms of track-width, wheelbase, and mass which is equally distributed over the volume of the vehicle. In order to perform the collision volume of the vehicle, we used a Unity BoxCollider that covered the complete body of the vehicle, being its center of mass selected from the geometric center of the vehicle chassis, as seen in Fig. 4. The four wheels were respectively controlled by four Unity WheelColliders, and their steering occurred in conformity with the Ackermann steering model. The control scripts set of the vehicle relied on [10], which was implemented via the RealisticCarControl V3 [44].

The parameters of the WheelCollider controlled the friction between the wheels and ground. The default settings of the friction parameters for the experiment are shown in Table 2. We also implemented a 9 degree of freedom Inertial Measurement Unit (IMU), GPS sensor, as well as longitudinal velocity. In addition, we monitored the steering angle of the vehicle as well as the side-slip angle. All these data were either directly received from Unity or calculated from the sensor data. Finally, we used two cameras with a resolution of 640 × 480 pixels as the front and rear cameras of the vehicle. The vertical field of view of the camera was 60∘, the horizontal field of view depended on the aspect ratio.

4.2Analysis of the image data

The foundation of this study is built upon an existing relationship between the data obtained from imaging sensors and the VSA. This approach is motivated by the acknowledged limitations of black-box machine learning models [45].

To develop a reliable model, we performed a comprehensive analysis of this relationship. This involved the collection and processing of image and VSA data from the simulated environment discussed above, conducting rigorous statistical tests to investigate correlations, and scrutinizing the relationship between the variables.

In each timestep t, we obtained βt∈ℝ, the image ℐ𝑓𝑟𝑜𝑛𝑡,t∈ℝR×C×3 from the front camera as well as ℐ𝑟𝑒𝑎𝑟,t∈ℝR×C×3 from the rear camera, where C is the number of columns of the image and R is the rows of the image.

In each timestep, we applied the GoogLeNet CNN [17] and obtained 1024 features for each image. In order to evaluate the relationship between these features and the VSA, we first standardized the features and extracted a de-correlated representation relying on the PCA [19]. We chose a variance-based method (PCA) since it offers a clear and intuitive understanding of the data’s structure. The principal components (lateral dimensions) derived from such methods are orthogonal, ensuring no redundancy, and they capture the directions of maximum variance in the data, making it more readable.

Figure 5.

The proposed method with a batch size of B, sliding window length of W and a output prediction horizon of P considering only one camera. The proposed approach utilizes various sensor data, including images (img), wheel angular velocities (ωx⁢y), steering angle (δ), longitudinal velocity (vx), longitudinal and lateral acceleration (ax and ay), yaw rate (ψ˙), as well as the derivative of the prediction target (β˙) computed using the kinematic single-track model, as input for time series prediction. The model integrates a CNN [17] for image-based feature extraction and a GRU with two recurrent layers, a hidden dimensionality of 128, and a tanh activation function for capturing temporal dependencies and ensuring effective information transfer across time steps. This amalgamation results in a hybrid kinematic CNN-GRU informed model that leverages exteroceptive data. The dimensionality of the various inputs, as well as the propagated values are displayed within square brackets.

The relationship between the first five principal components and the VSA was assessed using both Pearson’s and Spearman’s rank correlation tests. Pearson’s test was chosen as it effectively measures linear relationships between continuous variables providing insights into any linear correlation between the latent features and β. On the other hand, the Spearman’s rank correlation captures non-linear, monotonic relationships that might not be evident in a linear analysis. Together, these tests offer a comprehensive evaluation of both linear and rank-order correlations. The focus was primarily on the first five principal components of the CNN latent features. These components were chosen due to their significant contribution to the overall variance, thereby enhancing the effectiveness of the analysis.

We formulated two null hypotheses for our analysis.

Both hypotheses tested whether there was a significant relationship between the variables under consideration. If the p-value, the probability of observing the data or more extreme results under the assumption that the null hypothesis is true, was found to be less than the predefined significance level α of 0.001, then the null hypotheses will be rejected. We deliberately selected 0.001 as our significance level to impose a stringent criterion.

4.3Model implementation for VSA estimation

Our implemented prediction model consisted of a CNN, specifically GoogLeNet [17], for feature extraction on images, and a GRU, for time series prediction, as visualized in Fig. 5. The choice of GoogLeNet was motivated by its efficiency in feature extraction, computational affordability, and its ability to learn multi-scale features through inception modules [17], metrics that are crucial in various applications, including digital photogrammetry of piping systems [46]. These evaluation criteria align with the comprehensive computational exploration conducted by [47], who also emphasized the importance of methodological choices and feature extraction techniques. The chosen inputs consisted of a sliding window sequence W of a variety of sensor data typically provided via CAN BUS in a car (e.g. images (img), wheel velocities (ωx⁢y), steering angle (δ), longitudinal velocity (vx), longitudinal and lateral acceleration (ax and ay), yaw rate (ψ˙)). The GRU in our model is primarily employed to capture temporal dependencies and relationships in the data. It’s worth noting that while GRUs inherently transfer information across time steps in a sequence, this mechanism is distinct from the traditional concept of transfer learning, where pre-trained models are adapted for a different but related task. In the context of our model, the GRU’s architecture, characterized by its reset and update gates, ensures the effective transfer of relevant information from one time step to the next within the same task. This internal “transfer” of information across time steps is pivotal for the model to make informed predictions based on both historical and current data.

Furthermore, our research explored both sequence-to-one (S2O) and sequence-to-sequence (S2S) predictions. Specifically, S2O prediction involves processing a sequence of input data to produce a single output, while S2S prediction takes a sequence of input data and predicts a corresponding output sequence, allowing for more dynamic and temporally structured predictions [48]. The examination of different sliding window lengths provides critical insights into the temporal influence on the prediction accuracy of the model. We utilized the last feature map of a CNN [17] to provide exteroceptive information of the environment to the prediction model. We argue that the incorporation of exteroceptive information from image streams provides long term correction data and thus will improve the sequence-to-sequence prediction capabilities. Besides the aforementioned sensor data we utilized the kinematic single-track model to compute the derivative, of the prediction target which is fed to the linear layer via the motion model. For this we transformed the formula Eq. (6) to derive β˙, as seen in Eq. (7), by applying the assumption that cos⁡(β)≈1, and fed it as additional input to our GRU model thus creating a hybrid kinematic CNN-RNN model.

We argue that the proposed sensor suit is applicable for the use case due to the fact that (i) all new vehicles manufactured on or after May 1st, 2018 with a gross weight rating of 4536 kg or less are required to have a backup camera [49] and the general usage of cameras in vehicles increased by ≈ 315% from 2014 to 2020 [50], (ii) GPS, IMU and CAN data is available in modern cars [51, 52].

Figure 6.

Dataset metrics.

We implemented the model using PyTorch [53] and Cuda 12.1. To train and test our model we collected a total of 110 min data on snow covered asphalt in the 3DCoAutoSim simulator [54]. Table 3 depicts an overview of the dataset composition. For the initial hyperparameter estimation, we relied on 3-fold cross-validation with 20 epochs, Table 4 visualizes the results and selected parameters. Besides the initially chosen hyperparameters we utilized learning rate scheduling based on ReduceLRonPlateu11 to improve model performance and speed up the training process [55, 56]. Furthermore, to prevent overfitting we implemented early stopping when the validation loss did not decrease for 20% of the overall training epochs, L1-regularization (λ=0.001), and 20% neuron dropout. Finally, we performed a train-test split of 75%–25%, and a separate dataset was recorded for the final evaluation.

Besides the proposed approach shown in Fig. 5, we compared our model in S2O and S2S prediction to the approach implemented in [8]. In all cases we used a separate validation dataset to evaluate our model that was neither used during training nor testing. Since the frequency of the sensors differed, we re-sampled the collected data, in preprocessing, to a frequency of 10 Hz. We then performed outlier removal, by relying on the z-score of 3, which indicates that data points beyond 3 standard deviations from the mean were considered as outliers, and further prepared the dataset in a sliding window approach as depicted in Table 5. Furthermore, we employed a feature scaling technique to normalize the range of the independent variables in the dataset as part of the data preprocessing step. The resulting dataset consisted of 65919 sample points valid for training, testing, and evaluation. Figure 6 visualizes the distribution of the ground truth VSA over the complete dataset.

4.4Model evaluation

In light of the correlation analysis, the highly non-linear feature processing within our proposed CNN pipeline, and the distinct correlation of front and rear camera features with the VSA, we pursued two distinct experimental approaches. These approaches not only included image data but also leveraged other sensor information as previously described in this section.

Approach A utilized both front and rear image data in conjunction with the sensor data. This aimed to assess the combined impact of front and rear cameras and other sensor information on the model’s VSA prediction capabilities.

On the other hand, Approach B focused exclusively on the front image data and the sensor information, intentionally omitting the rear image data. Despite using the same dataset as Approach A, Approach B truncated the rear camera information. The rationale behind this strategy was to test the relevance and impact of the rear camera data as suggested by the correlation analysis.

4.4.2Sequence-to-sequence prediction

Given the interdependent nature of the VSA data, which dynamically evolves based on past, present, and future states, the S2S prediction paradigm is integral to our models, designed to capture and interpret the complex temporal dependencies that exist among different states in a sequence, thereby producing accurate estimations of future vehicle side-slip angles.

To evaluate the effectiveness of our S2S models, we conducted a series of comparative analyses using different prediction horizons. We employed the sSMAPE as our performance metric, providing a robust measure of the prediction accuracy of our models across various time horizons P={10,20,30,100}.

Figure 7.

Ground truth VSA of the evaluation dataset.

The outcomes yielded by our two experimental strategies in relation to the two distinct prediction paradigms, S2O and S2S, are delineated in Sections 5.2 and 5.3, respectively. To further fine-tune our models, we trained them using an array of hyperparameter combinations, these encompassed various sliding window input sizes W={10,20,30,50,100} and a spectrum of prediction horizons P={1,10,20,30,100}. The GT values of the evaluation dataset can be seen in Fig. 7.

Finally, we evaluated the model performance using the sMAPE, sequence wise sMAPE (sSMAPE) as well as the averaged sSMAPE (E¯).

5.1Correlation analysis

Table 6 reports the detailed findings of the correlation analysis described in 4.2, summarizing the estimated correlation values and their associated p-values for both Pearson’s and Spearman’s rank correlation methods. Each row represents one of the principal components from the PCA-based processed CNN features. Each column shows the estimate and p-value of the respective correlation method for both the front and rear cameras.

Figure 8.

Comparative differential plots of S2O predictions from Approach A (middle) and Approach B (bottom), showing deviation from ground truth. Different input window lengths of W=10 and W=100 are represented, highlighting the temporal influence on prediction accuracy.

5.2Sequence-to-one prediction model evaluation

Figure 8 showcases the effectiveness of our hybrid CNN-GRU model in executing the S2O prediction task. It presents comparative differential plots for both Approach A and Approach B under different sliding window lengths of W=10 and W=100. The divergence from the ground truth provides an insightful measure of the prediction accuracy of both approaches. The examination of these different sliding window lengths provides critical insights into the temporal influence on the prediction accuracy of the model. In our model training, we settled on 20 epochs based on preliminary experiments. Beyond this, we noticed an increase in validation loss, hinting at overfitting. While the ideal epoch count can vary with the dataset, for ours, 20 epochs ensured a balance between computational efficiency and model performance. As previously mentioned, the performance metric, sSMAPE, was employed to evaluate the model’s effectiveness in the S2O prediction paradigm. The calculated values are documented in Table 7, that provides a comparative overview of the S2O prediction outcomes of the various models. For a complete overview of the differential plots for all window sizes W={10,20,30,50,100}, refer to Appendix B, it provides a collection of plots for both Approach A (Fig. 10) and Approach B (Fig. 11), which encapsulate the variability of each models’ performance across a broad range of temporal sliding input windows.

Figure 9.

S2S sSMAPE comparison for each individual prediction horizon sequence.

In Fig. 8a and c we can see that, compared to the ground truth and the approach of [8], our models generated similar accurate results, except for cases when the target value is close to the min/max values of the training data, e.g. as seen at ≈ 180 ms. For larger input sizes, as seen in Fig. 8b and d, it is apparent that the deviation from the GT was significantly higher for all three models. The largest deviations of the GT for the models with W=100 were close to the min/max values of our training just like for the models with W=10.

5.3Sequence-to-sequence prediction model evaluation

Figure 9 presents the outcomes of applying the S2S prediction paradigm, illustrating the sSMAPE metric at each prediction horizon. For a more granular perspective see Table 8, as it displays the mean sSMAPEE¯ for respective prediction horizons P={10,20,30,100}. It visualizes the results of a comparative analysis of mean sSMAPE across varying prediction horizons, focusing on our proposed Approach A, Approach B, and the model established by [8]. The results reflect the performance of the S2S models in capturing and interpreting complex temporal dependencies existing among different states in a sequence, and accordingly, generating accurate estimations of future vehicle states.

The comparison of S2S averaged sMAPE for each individual prediction sequence for different horizons P between Approach A (Fig. 12) and Approach B is illustrated in the Appendix B (Fig. 13).

The comparison of the S2S prediction paradigms is demonstrated through Fig. 9, which illustrates the sSMAPE for each prediction horizon. The results for P=10 and P=100 at the initial prediction time step indicate that the model proposed by [8] consistently displays lower sSMAPE values than Approach A and Approach B in 9 out of 20 cases. Additionally, an increased window size correlates with an increment in the sSMAPE across all models. The last prediction step, however, exhibits a different trend with the baseline model of [8] scoring lower sSMAPE values only on 2 out of 20 prediction sequences.

When comparing Approach A with Approach B, results for the first prediction step show Approach A surpassing Approach B only in 2 out of 5 instances when P=100, specifically for window sizes of W=20 and W=30. A similar pattern is observed for P=10, where Approach A outperforms B only twice, again at window sizes of W=20 and W=30. Therefore, for the initial prediction step, Approach B holds the superior record 60% of the time.

For the last prediction step, the dynamics shift somewhat. Approach A surpasses Approach B in 4 out of 5 cases when P=100, spanning all window sizes except W=100. For P=10, however, Approach A does not outperform Approach B in any instance. Consequently, for the last prediction step, Approach A proves superior in 4 out of 10 cases.

Another point of comparison is the fluctuation of the sSMAPE over time. As depicted in Fig. 9a and c, our models generally exhibit an initial decrease in the sSMAPE which eventually increases after reaching a minimum. In contrast, the model of [8] holds a stable sSMAPE for the early time steps, followed by an increase. This observation suggests that both our models surpass the model of [8] in long-term predictions, with the exact intersection point varying based on the window size W. Specifically, for the window size W=100, both our models demonstrate a steadier sSMAPE as compared to the model of [8], which shows a continual increase of the sSMAPE across all window sizes except for W=100.

Finally, the models were compared based on their average sSMAPE (E¯) across various prediction horizons. Table 8 provided the mean sMAPE for the prediction horizons P={10,20,30,100} for Approach A, Approach B, and the model by [8].

6.1Correlation analysis

The results of the correlation analysis presented in Table 6 demonstrate several key insights regarding the relationship between PCA-based processed CNN features of the front and rear camera to β. A detailed investigation of these results is provided below.

High correlations are observed in the second, third, and fifth latent dimensions with β for the front camera, as demonstrated by Pearson’s correlation coefficients of 0.1951, -0.5244, and 0.2232, respectively.

These results are echoed by the Spearman’s rank correlation, suggesting that these statistical relationships are robust. The third latent dimension, in particular, shows a negative correlation, implying an inverse relationship with the VSA in this dimension. The first and fourth latent dimensions also display correlations significant at α=0.001, strongly refuting the null hypotheses H0,𝑃𝑒𝑎𝑟𝑠𝑜𝑛 and H0,𝑆𝑝𝑒𝑎𝑟𝑚𝑎𝑛.

The correlations for the rear camera present a different narrative, being notably lower across all latent dimensions compared to the front camera. Nevertheless, most latent dimensions still display statistical significance at α=0.001. An exception is found in the fourth dimension, which demonstrates a non-significant correlation in the Spearman’s rank correlation test, suggesting that this dimension’s relationship might not be robust or meaningful.

6.2Sequence-to-one prediction

Based on the results of the S2O prediction paradigm we recognize the need for a more in-depth exploration of our model’s capacity. We recognize that (i) additional data augmentation or an expanded training dataset may be necessary for superior data extrapolation. This realization emerged as we compared our approach to that of [8]. On the other hand, (ii) the divergence could also be rooted in the different methodologies for feature extraction from sensor data between our model and [8]’s. Furthermore, (iii) our model extends beyond the approach of [8] by incorporating exteroceptive sensor data, specifically image streams from vehicle cameras processed via CNN. Consequently, our CNN-GRU model captures a different spectrum of information for predicting the extremes of the VSA compared to a standalone GRU. This divergence may contribute to reduced accuracy near the extreme values of the training data. As we continue to refine our model, adjustments to the CNN and the inclusion of supplementary data sources are avenues we plan to explore for improving accuracy.

6.3Sequence-to-sequence prediction

Approach A seems to be slightly more beneficial, especially for window sizes W=20 and W=30. However, if the interest lies in the final prediction step with a horizon of P=100, Approach A might be preferred due to its superior performance in most instances. Another factor to consider is the stability of the sSMAPE over time, where both our models showcase a stronger performance than [8]’s model for larger window sizes. These observations suggest that our models may offer an advantage for applications requiring long-term predictions.

It is worth noting that Approach A surpassed Approach B in 13 out of 25 cases. This indicates that notwithstanding the diminished correlation of the rear camera features with the side-slip angle, these features nonetheless deliver consequential information that augments the prediction accuracy of our model. The origin of this discrepancy may be rooted in various elements. First and foremost, the nexus between the rear camera features and the side-slip angle could be non-linear and intricate, reducing its detectability by conventional correlation measures but can still be deciphered by state-of-the-art prediction models such as deep neural networks. The sophisticated architecture of such models enables them to extract latent, non-linear patterns in data, thereby enhancing prediction performance even in the absence of an evident linear relationship. Secondly, an overlooked aspect in the correlation analysis is the potential synergistic effect among different data types. In this scenario, the interaction of rear camera features with other pieces of information, such as front camera features or steering angle, could substantially enrich the predictive capacity of the model. This collective effect is not discernible by examining individual correlation coefficients, which provide a somewhat simplistic view of the relationship between individual variables and the target. Lastly, it is plausible that the rear camera features serve as particularly vital information in specific circumstances, for instance, when the vehicle is reversing or during a sharp turn. Such contextual utility of rear camera features contributes to a comprehensive, more accurate prediction across a variety of situations. Taken together, these observations highlight the importance of including diverse types of data in complex prediction tasks, even those with seemingly low correlation to the target variable. It underlines the multifaceted nature of data utility in model performance and cautions against over-reliance on individual correlation measures when making decisions about feature inclusion. The analysis thus suggests the potential benefit of leveraging machine learning models capable of capturing both linear and non-linear relationships and interactions among features.

Compared to the baseline model of [8] both of our models outperformed it for longer prediction horizons (⩾ 2 s). Especially Approach A demonstrated the best results for prediction horizons of two and three seconds, while Approach B outperformed Approach A on a prediction horizon of 10 seconds. This may be attributed to the model structure of Approach A that may be too complex for the little training data, leading to less accurate predictions. Therefore, Approach B, which is a smaller model, was better suited for longer prediction horizons as it was able to generalize better on the limited dataset.