Comparative study on the effect of color spaces and color formats on heart rate measurement using the imaging photoplethysmography (IPPG) method
Abstract
BACKGROUND:
The imaging photoplethysmography (IPPG) technology has been demonstrated to be an effective method for heart rate (HR) monitoring. However, some interference caused by the ambient illumination variation and facial motion severely influences the accuracy of the HR measurement. Some color spaces and color formats are assumed to reduce the interference, and enhance the accuracy of HR estimation.
OBJECTIVE:
The aim is to identify the optimal color space and format for IPPG based HR measurement.
METHODS:
Six color spaces and 3 color formats are compared in this study, based on an IPPG based HR measurement system. 424 pieces of videos captured by the system are used for the selection of the optimal color channel and color space; while 10 pieces of videos are for the identification of the optimal color format.
RESULTS:
The results shows that the green channel of RGB space is the optimal color channel, and RGB is the optimal color space, in respect of the mean squared error of HR estimation. BayerBG 8bit is found to be the optimal color format for video recording, which can significantly reduce the HR estimation error.
CONCLUSIONS:
BayerBG 8bit color format for video recording, and RGB color space for video analysis is suggested for the IPPG based HR measurement system. The suitable configuration of color space and format could enhance the accuracy of HR measurement.
1.Introduction
Heart rate (HR) is an important physiological parameter for the assessment of physical health status, and is also a critical indicator for cardiovascular diseases [1, 2, 3, 4, 5, 6]. In recent years, the imaging photoplethysmography (IPPG) method that could extract HR from the facial color variation information, has achieved rapid developments [7, 8]. Compared with traditional contact measurement methods, IPPG does not require the subject to wear any sensors, therefore it showed significant advantages in some scenarios, such as neonatal monitoring, monitoring during haemodialysis, and fatigue driving judgment. Family healthcare also benefits a lot from IPPG technology, especially with the rapid dissemination of smartphones [9].
The contact-free IPPG method with a camera was firstly proposed by Verkruysse et al., and showed that IPPG signals have different intensity levels in RGB channels [7]. The phenomenon indicated that the IPPG signals containing the pulsatile information distributed in specific wavelength ranges, and expressed as components with different intensity in the 3 channels of RGB videos. Because the reflected light from facial tissue in the wavelength from 541 to 577 nm was more sensitive to the pulsatile blood volume, the green (G) channel of RGB videos was assumed to be the optimal channel for HR estimation in RGB format [10].
To extract the components associated with pulsatile diffusion of facial tissue from all the channels of RGB videos, conventional blind source separation (BSS) technologies such as independent component analysis (ICA) were originally designed. When considering multiple color channels of RGB videos, ICA could separate the IPPG component from other noises, which were assumed to be independent to IPPG component [11]. Lewandowska et al. proposed another BSS method for measuring HR based on the principal component analysis (PCA), and assessed the effectiveness of the method for different regions of interest (ROI), different combinations of color channels, and different lightening conditions [12]. His study showed that the accuracy of the method was affected by the variable of ambient illuminations. Therefore, some researchers began to focus on the methods to reduce the influence of ambient illumination, and found that some color spaces could effectively reduce the influence and therefore enhance the accuracy of HR estimation. Haan et al. built two orthogonal chrominance signals in RGB color space based on the dichromatic reflection model, and used the difference of the two signals to eliminate the artifact caused by the specular reflection component [13]. An improved motion robustness of IPPG method based on the signal of the normalized blood-volume pulse vector in a normalized RGB color space was also proposed by Hann [14]. Bal also proved that the IPPG method based on the two orthogonal vectors of RGB color space could reduce artifacts, employing a dual tree complex wavelet transform based denoising algorithm [6].
Besides RGB cameras, some multi-channel cameras were also used to improve the effect of IPPG based HR estimation. Mcduff et al. designed an experiment to systematically study the influence of illumination, motion, skin tone, and distance variance on the accuracy of heart rate measurement, and found that a 5-channel camera might significantly eliminate these interference factors [15]. But the high price of the multi-channel camera undoubtedly limited the application of this method. More researches tended to develop a denoising method based on a RGB formatted camera. Therefore, many color spaces, such as YCbCr, HSI, and LAB, were introduced into the denoising methods of IPPG signals [16, 17]. These color spaces converted from RGB, were considered as a convenient method to separate the luminance signal, which might contribute to enhance the accuracy of HR estimation. Zhang et al. proposed that LAB color space could remove the stationary noise and motion artifacts, and therefore could significantly reduce the error of HR estimation [18].
Although many color spaces have been introduced into the study of IPPG method, there is little study on the identification of the optimal color channel or color space under the same experimental condition. Moreover, the color spaces studied before are all converted from RGB color format; while other color formats for video recording such as BayerBG have never been considered during the past IPPG studies. In this paper, a framework for HR measurement is developed to reveal the effect of color spaces and color formats on the accuracy of IPPG based HR estimation. The conversion from RGB to other color spaces is firstly performed, in which the signal of the color channel is analyzed in frequency domain to extract HR. Comparing the accuracy of HR estimation, the optimal color channel and color space is identified with the dataset captured by our IPPG measurement system. Based on this measurement system, the effect of the color formats are also studied to identify the optimal color format for video recording. This study can optimize the color format and color space during the video recording and analysis of IPPG method.
2.Method
2.1Color spaces
Several studied have suggested that the reflected light from the facial skin of the object influences the accuracy of the IPPG based HR estimation method. The reflected light collected by the camera includes 2 components: the surface reflection without any pulsatile information, and the diffuse reflection from the skin tissue, which manifests the pulsatile blood volume in the skin tissue. A suitable color space may extract the diffuse reflection information from the reflected light color, therefore enhance the accuracy of HR estimation.
Figure 1.
The main color spaces used for IPPG analysis.
The common used color spaces for IPPG analysis include RGB, HSI, HSV, YIQ, YCbCr, and LAB (Fig. 1a to f). These color spaces are different types of color modes, used in image processing system for various purpose. They all describe the color in a 3D space, which consists of 3 independent attributes. But since the attributes used to describe the color are different, they build into different color spaces. Here we introduce the color spaces used in this comparative study.
RGB color space, that is defined according to the mechanism how human eyes recognize colors, is the most used color space in image processing. The RGB space includes 3 parameters, namely red (R), green (G), and blue (B), which can describe most colors. The value of the parameter, i.e. the channel, reflects the brightness of each color. When the values of 3 channels are all 0 (point coordinate was (0, 0, 0) in RGB space), the color is black; while it represents the white color when the 3 values are all 255.
In HSI color space, H (Hue) represents the wave length of the color; S (Saturate) represents the darkness of the color; I (Intensity) represents the brightness [19]. The HSI color space is built in a bipyramid shape. The value of H ranges from 0
(1)
(2)
(3)
HSV color space is a simplified form of HSI, in which,
(4)
(5)
In YIQ color space, Y represents the brightness; I represents the change from the orange to the cyan color; Q represents the change from the purple to the greenyellow color. YIQ color space is adapted to the image series with different brightness; therefore, it is usually used in the processing of color images. The transformation from RGB to YIQ is as follows:
(6)
YCbCr color space is widely used in digital camera system or video processing. The 3 parameters used in this color space includes: Y, which represents the brightness, Cb, which represents the blue component, and Cr, which is the red component. The transformation from RGB to YCbCr is as follows:
(7)
LAB color space is similar to YCbCr, which consists of a brightness component, and two chroma components. In this color space, L represents brightness, A represents the change from the green to the red color, B represents the change from the yellow to the blue color. LAB is considered as an expanded space of RGB, and the value of each channel in LAB can be calculated from RGB channels as [18]:
(8)
(9)
(10)
2.2HR estimation experiment
During most imaging systems, the camera captures the facial images, and records them as RGB format. Therefore, all the color spaces above are calculated from RGB channels. To evaluate the accuracy of HR estimation with the color spaces mention above, we developed an IPPG based HR measurement system, with a camera that recorded the facial video in RGB format. The experiment of HR measurement was conducted indoor with natural light, using the HR measurement system (Fig. 2). The subject who signed the informed consent, was asked to sit on the chair before the camera at the distance of 1.5 m. The subject was allowed to move lightly and breathe freely during the measurement. A single-lead ECG recorder was used as a golden standard for HR measurement. Two electrodes were placed at the up and down edges of the left ectopectoralis. The ECG signal captured from the electrodes was detrended, and then analyzed in frequency domain. The HR frequency was identified by the frequency where the energy peak appeared from 0.8 to 4 Hz in the frequency domain.
Figure 2.
The scheme of the experiment.
The RGB formatted facial video was converted to other color spaces, including HSI, HSV, YIQ, YCbCr, and LAB. The videos in different color spaces were processed as Fig. 3 shows. In the first processing method, each channel of the color space was selected, and processed with averaging smoothing, detrending, and whitening to void the interference caused by the variation of natural light and other noise. Then, the signal of the channel was transformed into frequency domain to identify the energy peak during 0.8–4 Hz. The corresponding frequency to the energy peak was considered as the pulsatile frequency, sixtyfold of which equaled the estimated HR. The estimated HR was then compared with the value measured by ECG, to calculate the mean squared error (MSE) of the estimation with the channel. This processing way was to compare the channels of color spaces, to identify the optimal channel for IPPG based HR estimation.
Figure 3.
The processing flowchart to extract HR from facial videos with different color spaces..
Another processing way is to compare the effect of color spaces on HR estimation. After the averaging smoothing, detrending, and whitening processing, the 3 channels of each color space were conducted independent component analysis (ICA) to generate 3 components, in which the component with the highest association with the optimal channel selected above was considered as IPPG component. IPPG component was transformed into frequency domain to extract HR exactly like the first processing way.
Figure 4.
BayerBG pixel location.
2.3Color format
It should be noticed that RGB channels are not the original output of the camera, because the camera usually needs an interpolation computation from the raw data to generate RGB channels. One of the common format for raw data is BayerBG format. Although the color space of the whole image is the same as RGB format, there is only one color channel for one pixel. The pixels with R, G, and B channels are interlaced like Fig. 4. Actually, all RGB images are obtained from BayerBG format, in which the R, G, and B channels around each pixel are interpolated to get the RGB values for the pixel, as follows:
(11)
(12)
(13)
Although RGB and BayerBG color formats were both supported in many imaging systems, most IPPG studies used RGB format for video recording. BayerBG was seldom used in IPPG studies to evaluate the effect of this color format on HR estimation. Therefore, we developed a software on C# platform that was able to record facial videos in both RGB and BayerBG formats, to compare which color format may obtain higher accuracy in HR estimation. The graphic user interface (GUI) of the software is shown in Fig. 5. When measuring HR with the system, the software connected the camera, and fed back the information of the camera at the upleft textbox. Then, a preliminary acquisition of the video was performed to show the video acquisition effect at the right panel. The ROI was automatically located at the forehead of the subject. When pressing the “Measure” button, the HR would be displayed at the left bottom within about 10 seconds. The HR estimation method was introduced in the previous section. The frame rate could be adjusted according to the brightness of natural light. The color format could be selected as BayerBG 8 bit, BayerBG 10 bit, or RGB format. BayerGB 8 bit recorded videos in BayerGB format, in which the value of each channel occupied 8 bit for each pixel, namely ranged from 0 to 255. While in BayerGB 10 bit format, the value of each channel occupied 10 bit for each pixel, namely ranged from 0 to 1023.
Figure 5.
The software of the HR measurement system. The right panel shows the video captured by the camera, in which the red rectangle shows the ROI at the forehead. The HR was displayed at the left-bottom. The color format could be adjusted to BayerBG 8bit, BayerBG 10bit, or RGB format.
2.4Statistical analysis
To obtain the optimal channel for HR estimation, we compared the HR estimation results from each channel of 6 color spaces, including RGB, HSI, HSV, YIQ, YCbCr, and LAB, with the HR value measured by ECG, to calculate MSE. The channel with the least MSE could get one score for a piece of video. The channel that obtained the highest score, was selected as the optimal channel. We captured 424 pieces of 10 second videos for the selection of the optimal channel, and also for the identification of the optimal color space.
To identify the optimal color space for HR estimation, the HR estimated from ICA results of each color space was compared with the golden standard from ECG. The color space that obtained the least MSE, was considered as the optimal color space for HR estimation.
We also compared the effect of color formats on the accuracy of HR estimation. The HR estimated from the videos with RGB, BayerBG 8 bit, and BayerBG 10 bit, was compared with the golden standard from ECG by Bland-Altman analysis. The MSE of HR estimation was also compared with student t-test between the three formats. The linear regression was performed between the HR values measured by ECG and IPPG methods with different color formats, to compare the association and linearity of the results for HR measurements. Ten pieces of 180 second videos were captured for each format, and used for the statistical analysis.
3.Results
3.1The optimal color channel and color space
424 pieces of 10 seconds videos were captured by our HR measurement system from 200 Chinese subjects. In order to identify the optimal color channel and color space, each video was processed with the methods stated above. The time consumption of the video processing and HR calculation was less than 1 second on our system (CPU: Intel i7-8650U, RAM: 16 GB). All these videos were captured with RGB format, and then converted into HSI, HSV, YIQ, YCbCr, and LAB color spaces. We compared the effectiveness of all channels in different color spaces in the HR estimation. As Fig. 6 shows, G channel of RGB color space had the highest score, meaning that G channel had the least MSE for HR estimation in most cases. Besides, H channel in HSI color space, Cr channel in YCbCr color space, and A channel in LAB color space, also got high scores, meaning that these channels was also good choices for IPPG based HR analysis.
Figure 6.
The score for the optimal channel among all channels in different color spaces.
Figure 7.
The MSE of different color spaces. The MSE was calculated by comparing the heart rate measured by IPPG method and ECG.
We also compared the effectiveness of color spaces in HR estimation by calculating HR estimation error. Figure 7 shows that HR estimation in RGB space was the best among the 6 color spaces, with the error of 2.868 bpm. The HR estimation error of LAB was a little bigger than RGB. But other color spaces had much bigger errors, indicating that these color spaces were not suitable for HR estimation.
3.2The optimal color format
The optimal color format for video recording during IPPG based HR estimation was identified with 10 pieces of 180 second videos from 10 Chinese subjects. The HR estimation results from the 3 color formats, including RGB, BayerBG 8 bit, and BayerBG 10 bit, were compared with HR measured by ECG with Bland Altman analysis. The results for Bland-Altman analysis were shown in Fig. 8, in which the bias and standard deviation of HR estimation was also shown. We found that the bias of HR estimation with BayerBG 8 bit color format was least, indicating that BayerBG 8 bit color format could reduce the system error in IPPG based HR estimation. The standard deviation was least in BayerBG 10 bit format, suggesting that the BayerBG 10 bit format could reduce the random error of HR estimation.
Figure 8.
The Bland-Altman analysis results for comparing the HR measured by ECG and IPPG with different color formats. (a) BayerBG 8 bit; (b) BayerBG 10 bit; (c) RGB.
Figure 9.
The MSE of HR measurement with different color formats. Bayer8 and Bayer10 represent BayerBG 8 bit and BayerBG 10 bit color formats, respectively.
The MSE of HR estimation with the 3 color formats is shown in Fig. 9. The error of HR estimation from images of BayerBG 8 bit format was significantly lower than the error of BayerBG 10 bit and RGB format, indicating that BayerBG 8 bit was the most suitable color format for HR estimation. Comparing to the BayerBG format, RGB format had a bigger MSE, validating our hypertension that BayerBG could be a better color format than RGB for IPPG based HR estimation.
Figure 10.
The linear regression between the HR values measured by ECG and IPPG with different color formats. (a) BayerBG 8 bit; (b) BayerBG 10 bit; (c) RGB.
The linear regression results also showed that BayerBG 8 bit was the most suitable color format for HR estimation, because the HR calculated from BayerBG 8 bit had the highest linear association with the HR measured by ECG (Fig. 10). The association between the HR values measured by ECG and RGB formatted IPPG was lower than the 2 types of BayerBG formats, suggesting that BayerBG formats could enhance the accuracy of HR measurement in a large range of HR.
4.Discussion
We developed an IPPG based HR measurement system to provide a platform that might reveal the effect of color spaces and color formats on the accuracy of HR estimation. Because all the videos used in this paper were captured by this system, all the color spaces and formats were compared under the same condition. Although there were some studies on the effect of color spaces on HR measurement, the number of their samples (usually 20–90 samples) were much less than our study. Moreover, the HR measurement conditions, such as the light environment, the duration of videos, the acquisition parameters of the camera, varied largely in their studies. Therefore, our study might provide a systematic comparison between the common-used color spaces and formats.
The optimization of the color space used in IPPG method is an interesting and valuable work, because it may enhance the accuracy of HR measurement without increasing the cost of cameras. In this paper, we used two methods to process the video for HR estimation, one without ICA, and the other with ICA. The method without ICA was used to select the optimal channel for all color spaces; while the method with ICA was for the identification of the optimal color space. Our results showed that a suitable color space might separate the channel with IPPG signal from other noises, like the G channel in RGB space. But also, a suitable color space (RGB) might enhance the effect of ICA results, extracting the independent component with IPPG signal with BSS.
Although there are still some controversies about the optimal color channel and space for IPPG method, lots of studies have suggested that RGB color space is a more effective one under most experimental conditions. In 2021, Zhang et al. proposed that LAB color space may effectively reduce the estimation error of HR. But the comparative study between RGB and LAB color spaces was still lacking. We compared RGB and LAB color spaces in this paper, with our HR measurement system. The results showed that, although LAB performed well in HR estimation, RGB was still the optimal color space for IPPG based HR estimation.
The color format of the video recording is also an important factor for HR measurement, however there is limited studies on its effect. We firstly developed a HR measurement system, which supported both RGB and BayerBG formats. Our study revealed the effect of color formats on the accuracy of HR estimation, and showed that BayerBG 8 bit was a better color format than RGB for video recording in light of HR estimation. Although BayerBG 10 bit also showed a lower random error than RGB, the bias and linearity of HR measurement was not as good as BayerBG 8 bit. The result is interesting, because BayerBG 10 bit consumes more bits for the recording of each pixel, but increases the system error for HR measurement. The reason may be that, the BayerBG 10 bit puts forward higher requirements for HR measurement conditions, such as the light environment, and the stable voltage supply for the camera; otherwise, this color format may increase the quantization error during the video recording.
Another important factor for HR measurement is the skin color of subjects, which was the limitation of the present study. The subjects are all Chinese, without heavy makeup or any tattoo on faces. The accuracy of our HR measurement system in the subjects with various skin colors might be different. The tattoo and scars on faces may also influence on the HR measurement. Although Mishra suggested that the variability of skin color had only limited effect on the accuracy of video-based HR measurement (with the error within 2 beat/min), the sample size in his study was not big enough to validate his opinion [21]. To clarify this issue, a large number of subjects with various skin colors were needed for the further study.
5.Conclusions
In this paper, a new platform for IPPG based HR measurement is proposed and demonstrated. Based on this platform, the effect of color spaces and color formats on the accuracy of HR estimation is studied. 424 pieces of videos are used for the identification of the optimal color channel and color space for HR estimation, and 10 pieces of videos are used for the identification of the optimal color format. Experimental results show that the HR measurement using RGB color space is in good agreement with the ground truth of HR with the MSE of 2.868 bpm, while the optimal channel for HR estimation is the G channel in RGB space. For the color format of video recording, BayerBG 8 bit is proved to be the optimal format, and significantly reducing the MSE comparing with RGB format. Therefore, BayerBG 8bit color format for video recording and RGB color space for video analysis are suggested for the following IPPG based HR measurement in practice.
Acknowledgments
The study is supported by the National Natural Science Foundation (No. 11672026), the State Key Laboratory of Virtual Reality Technology and Systems, and the Beijing Academy of Science and Technology Budding Project (No. BGS201913).
Conflict of interest
None to report.
References
[1] | Bruser C, Antink CH, Wartzek T, Walter M, Leonhardt S. Ambient and unobtrusive cardiorespiratory monitoring techniques. IEEE Reviews in Biomedical Engineering. (2015) ; 8: (1): 30-43. |
[2] | Fox K, Borer JS, Camm AJ, Danchin N, Ferrari R, Sendon JLL, Stg PG, Tardif JC, Tavazzi L, Tendera M. Resting heart rate in cardiovascular disease. Journal of the American College Cardiology. (2007) ; 50: (9): 823-830. |
[3] | Chen X, Cheng J, Song R, Liu Y, Ward R, Wang ZJ. Video-based heart rate measurement: recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement. (2019) ; 68: (10): 3600-3615. |
[4] | Huo C, Xu G, Li W, Xie H, Zhang T, Liu Y, Li Z. A review on functional near-infrared spectroscopy and application in stroke rehabilitation. Medicine in Novel Technology and Devices. (2021) ; 11: (1): 100064. |
[5] | Kranjec J, Begus S, Gersak G, Drnovsek J. Non-contact heart rate and heart rate variability measurements: a review. Biomedical Signal Processing and Control. (2014) ; 13: (1): 102-112. |
[6] | Bal U. Non-contact estimation of heart rate and oxygen saturation using ambient light. Biomedical Optics Express. (2015) ; 6: (1): 86-97. |
[7] | Verkruysse W, Svaasand LO, Nelson JS. Remote plethysmographic imaging using ambient light. Optics Express. (2008) ; 16: (26): 21434-21445. |
[8] | Poh MZ, Mcduff DJ, Picard RW. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express. (2010) ; 18: (10): 10762-10774. |
[9] | Scully C, Lee J, Meyer J, Gorbach AM, Granquist-Fraser D, Mendelson Y, Chon KH. Physiological parameter monitoring from optical recordings with a mobile phone. IEEE Transactions on Biomedical Engineering. (2012) ; 59: (2): 303-306. |
[10] | Chen DY, Wang JJ, Lin KY, Chang HH, Wu HK, Chen YS, Lee SY. Image sensor-based heart rate evaluation from face reflectance using Hilbert-Huang transform. IEEE Sensors Journal. (2015) ; 15: (1): 618-627. |
[11] | Poh MZ, Mcduff DJ, Picard RW. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering. (2011) ; 58: (1): 7-11. |
[12] | Lewandowska M, Nowak J. Measuring pulse rate with a webcam. Journal of Medical Imaging and Health Informatics. (2012) ; 2: (1): 87-97. |
[13] | de Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering. (2013) ; 60: (10): 2878-2886. |
[14] | de Haan G, van Leest A. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiological Measurement. (2014) ; 35: (9): 1913-1926. |
[15] | Mcduff D, Gontarek S, Picard RW. Improvements in remote cardio-pulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering. (2014) ; 61: (10): 2593-2601. |
[16] | Nooralishahi P, Loo CK, Shiung LW. Robust remote heart rate estimation from multiple asynchronous noisy channels using autoregressive model with Kalman filter. Biomedical Signal Processing and Control. (2019) ; 47: (1): 366-379. |
[17] | Hassan MA, Malik AS, Fofi D, Saad N, Karasfi B, Ali YS, Meriaudeau F. Heart rate estimation using facial video: a review. Biomedical Signal Processing and Control. (2017) ; 38: (1): 346-360. |
[18] | Zhang Y, Dong Z, Zhang K, Shu S, Lu F, Chen J. Illumination variation-resistant video-based heart rate monitoring using LAB color space. Optics and Lasers in Engineering. (2021) ; 136: (1): 106328. |
[19] | Hsieh IS, Fan KC, Lin CH. A statistic approach to the detection of human faces in color nature scene. Pattern Recognition. (2002) ; 35: (7): 1583-1596. |
[20] | Herodotou N, Plataniotis KN, Venetsanopoulos AN. Automatic location and tracking of the facial region in color video sequences. Signal Process: Image Communication. (1999) ; 14: (5): 359-388. |
[21] | Mishra S, Tiwary US. Heart rate measurement using video in different user states for online HCI applications. Procedia Computer Science, (2014) ; 39: (1): 20-27. |
