Severity detection of COVID-19 infection with machine learning of clinical records and CT images

1.Introduction

Coronavirus disease 2019 (COVID-19) is a deadly viral infection that has spread rapidly around the world since its outbreak in 2019. According to a report from the World Health Organization, as of 25 May 2022, there have been more than 524 million confirmed cases of COVID-19, including more than 6.28 million deaths 0. The severity of COVID-19 can be classified into the following categories: mild, ordinary, severe, and critical [2]. Patients with severe COVID-19 may suffer from massive alveolar damage and respiratory failure, leading to death [3]. Therefore, early classification of COVID-19 and effective targeted treatment for critically ill patients can reduce the risk of complications. Early and automatic diagnosis will help countries all over the world provide timely treatment and quarantine. Hospitals can also offer more professional treatment for severe COVID-19 patients.

Nucleic acid screening, clinical features, epidemiological features, and imaging findings are basic for diagnosing COVID-19 [4]. After comparing the diagnostic effects of RT-PCR tests and chest computed tomography (CT) on the initial negative to positive diagnosis, AI [5] concluded that chest CT detection is faster. Chest CT is recommended as a routine test for surveillance and diagnosis of COVID-19 due to imaging features such as ground-glass opacity and consolidation on Chest CT used to determine SARS-CoV-2 infection-associated pneumonia [6]. A chest CT also assists physicians in identifying the early stages of lung infection [8] and is beneficial in helping governments to establish greater public health surveillance and response systems [10]. A combined assessment using clinical records and imaging features allows for a more accurate early diagnosis of patients with COVID-19.

Although some studies have constructed machine learning (ML) prediction models, to our knowledge, the current diagnosis of COVID-19 does not achieve satisfactory accuracy. Terwangne et al. [11] used 295 RT-PCR-positive COVID-19 patients data to develop a model based on the Bayesian network to predict the severity grading of COVID-19 patients and finally obtained a AUC of 83.8%. Yao et al. [12] developed a model to predict the severity of COVID-19 based on the SVM algorithm using data from 137 COVID-19 patients (75 severe, 62 mild) and finally obtained an accuracy of 81.48%.

Liang et al. [13] developed a model for predicting the severity of COVID-19 patients based on the SVM algorithm using data from 172 patients (60 severe) and finally obtained an average accuracy of 91.83%. Liang et al. [14] used data from 1590 (131 severe) patients to develop a model based on the LR algorithm for predicting clinical risk scores for the occurrence of critical illness in hospitalized patients with COVID-19, ultimately obtaining an AUC of 88% on the validation set.

Zhu et al. [15] used data from 127 patients (16 severe groups) to develop a model to assess the severity of infection in COVID-19 patients based on the LR algorithm, ultimately obtaining an AUC of 90.0%.

The above studies related to the severity classification of COVID-19 were based on clinical features only. They did not consider the influence of CT image feature factors on the classification of mild to severe disease. Because of this, we use clinical examination data and CT radiomics features to build a prediction model for mild and severe COVID-19 patients. The study aims to segment the region of interest in the lung, extract CT radiomics features based on the segmentation results, and train a machine learning prediction model using the clinical records and the extracted image features. In the following sections, data collection and modeling methods will be described first, followed by developing COVID-19 patient severity detection models using traditional and ensemble machine learning algorithms.

2.Materials and methods

The clinical and chest CT data used for this retrospective study were collected by the Shanghai Public Health Clinical Center from 20 January to 29 May 29 2020. The Ethics Committee approved the retrospective study of the Shanghai Public Health Clinical Center. Figure 1 shows the overall flowchart we developed for detecting the severity of COVID-19. Firstly, SP-V-Net was used to segment the CT images to obtain the lung contours. Radiomics was used to extract the lung image features, which were combined with clinical features to build a mild-severe diagnostic model after feature screening.

The 427 clinical data collected were used for this study, including 387 patients with mild COVID-19 (mean age, 40.26 ± 15.16 years; range, 22–88 years; male 213, female 174), 40 subjects with severe COVID-19 (mean age, 60.58 ± 14.69 years; range, 23–73 years; male 28, female 12), and seven eventually died. The relevant scan parameters for chest CT are as follows: slice number, 49–90; matrix, 512 × 512; pixel size, 0.590–1.0 mm; slice thickness, 5 mm. In the next diagnostic task of COVID-19 severity, all subjects were divided into mild and severe COVID-19 groups for binary classification. The baseline characteristics of patients with mild and severe COVID-19 are shown in Table 1.

Figure 1.

Flowchart of our approach to building the COVID-19 severity diagnosis model.

By observing the basic situation of the clinical dataset, we found that some clinical features had missing data, and the number of missing data and the proportion of missing data for different features are shown in Table 2. To reduce the impact of missing data on the experimental results, we use the Multivariate Imputation by Chained Equation (MICE) [16] method to interpolate the missing data. MICE is a multiple interpolation method that works iteratively and can resolve the uncertainty of missing values by creating multiple interpolations. The data can be interpolated by variable by specifying an interpolation model for each variable. The MICE interpolation technique has better robustness and better accounts for uncertainty and is selected to treat missing data values.

2.5Model training and evaluation

3-fold cross-validation was used to prove the performance of the model on the training data set. The training data consist of 67% of the total data; the test data consist of the remaining 33%. Among all 427 sets of patient data, 285 (258 mild, 27 severe) data sets were used as training data, and 142 (129 mild, 13 severe) data sets were used as test data. Finally, we used training data to train eight different machine learning algorithms and the trained models to predict the test data. Accuracy, f1-score, AUC, and AUCPRC were used to analyze the model’s performance. Figure 2 shows the workflow of our method.

Figure 2.

Overall training flowchart of the COVID-19 mild-severe disease prediction model.

To better measure the model’s effectiveness, we used different evaluation indicators to compare the multiple aspects. According to the confusion matrix, the following indicators were used to evaluate the performance of the model comprehensively:

TP, FP, TN, and FN stand for True Positive, False Positive, True Negatives, and False Negative respectively.

(3)

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦=T⁢P+T⁢NT⁢P+F⁢P+F⁢N+T⁢N

(4)

𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛=T⁢PT⁢P+F⁢P

(5)

𝑟𝑒𝑐𝑎𝑙𝑙=T⁢PT⁢P+T⁢N

(6)

F1-score=2×(𝑟𝑒𝑐𝑎𝑙𝑙×𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛)𝑟𝑒𝑐𝑎𝑙𝑙+𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛

(7)

G-mean=𝑟𝑒𝑐𝑎𝑙𝑙⋅𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛

(8)

𝑀𝐶𝐶=T⁢P×T⁢N-F⁢P×F⁢N(T⁢P+F⁢P)⁢(T⁢P+F⁢N)⁢(T⁢N+F⁢P)⁢(T⁢N+F⁢N)

(9)

𝐴𝑈𝐶𝑃𝑅𝐶=Area Under precision-recall Curve

(10)

𝐴𝑈𝐶=area under the curve

3.Results

3.1Feature selection

To further verify the effect of different features on machine learning prediction results, we performed 100 RandomForest feature sorting on clinical features, CT radiomics features, a mixture of clinical features, and CT radiomics features, respectively. The final mean value of 100 experimental results was calculated as the final feature importance ranking result. We further filtered the number of features that make different machine learning algorithms get optimal results based on the feature ranking results. The optimal number of feature combinations for different machine learning models is shown in Fig. 3.

Figure 3.

Number of features for each machine learning algorithm to obtain optimal prediction results.

Figure 4.

Experimental results of AUCPRC obtained by various machine learning algorithms using different types of features.

The order of feature selection was sorted according to the score of the random forest algorithm from high to low. The feature importance scores of the clinical features and the CT extracted features are shown in Table 3. To get the best prediction results, we used gridSearchCV to optimize the parameters of each machine learning model. The optimal model parameters are shown in Table 4.

Table 3

Feature importance scores

Feature name	Feature importance score
PO2	0.0513
Age	0.0264
original_firstorder_Entropy	0.0211
original_glcm_SumEntropy	0.0189
PCT	0.0180
original_firstorder_RobustMeanAbsoluteDeviation	0.0170
Lactate dehy drogenase	0.0168
C-reactive protein	0.0166
original_glrlm_GrayLevelNonUniformityNormalized	0.0164
original_firstorder_Kurtosis	0.0159
original_firstorder_InterquartileRange	0.0150
original_glrlm_RunEntropy	0.0150
original_glcm_JointEnergy	0.0147
APTT	0.0143
original_glcm_JointEntropy	0.0142
original_firstorder_Skewness	0.0140
original_firstorder_Uniformity	0.0135
original_glszm_SmallAreaEmphasis	0.0126
original_glcm_Id	0.0126
eGFR	0.0125
original_glcm_InverseVariance	0.0125
Na	0.0123
original_glszm_SizeZoneNonUniformityNormalized	0.0119
Total cholesterol (TC)	0.0115
original_glszm_ZoneEntropy	0.0104
original_glszm_GrayLevelNonUniformityNormalized	0.0098
original_glszm_ZonePercentage	0.0095
original_firstorder_MeanAbsoluteDeviation	0.0093
original_glcm_MaximumProbability	0.0092
original_glcm_Idm	0.0086

Table 4

The optimal model parameters

Algorithm	Model parameters	Value
KNN	n_neighbors	11
	p	1
LR	Penalty	L2
	C	1
	class_weight	Balanced
RF	class_weight	Balanced
	Criterion	Entropy
	max_depth	14
	min_samples_leaf	1
	n_estimators	65
XGB	learning_rate	1
	n_estimators	64
	max_depth	2
	subsample	0.8
	reg_alpha	0.1
	reg_lambda	1
GBDT	boosting_type	gbdt
	colsample_bytree	0.6
	learning_rate	1
	n_estimators	32
	num_leaves	50
	objective	Binary
	reg_alpha	0.1
	reg_lambda	10
	subsample	0.8
MLP	hidden_layer_sizes	3
	Activation	Identity
	Solver	lbfgs
	learning_rate	Constant
	random_state	2008

After parameter optimization and combining the optimal features, we trained the COVID-19 mild-severe prediction models with eight machine learning algorithms. Due to some imbalance in the experimental data, we finally chose the PRAUC value to verify the model’s goodness. Ten 3-fold cross-validation experiments were conducted separately, and the mean values were obtained as the final results of the experiments. Figure 4 shows the experimental results of AUCPRC obtained by each machine learning algorithm using different types of features. By comparing the experimental results, we can see that the final results obtained by different machine learning algorithms using the combined Clinical and CT radiomics features are higher than those obtained with clinical or CT radiomics features alone. The experimental results also further demonstrate the more accurate results of the machine learning classification model built using clinical and CT radiomics features. Therefore, to further verify the effect of data imbalance on the experimental results. We will select combined clinical and clinical feature data for further sampling experiments in the following experiments.

3.2Evaluation of diagnostic performance

In the following data sampling experiments, 3-fold cross-validation was used to divide the data, and then the divided data were combined with different sampling methods for the sampling experiments. The Under-sampling methods were used to sample 0.2, 0.4, 0.6, 0.8, and 1.0 times the majority of the data. The Over-sampling methods were used to sample 0.1, 0.2, 0.4, 0.6, and 0.8 times the majority of the data for the minority data. All sampling methods were performed on the training set data, and no processing was done on the test set data. Ten 3-fold cross-validation experiments were conducted separately, and the average of the ten experimental results was used as the final result. Figure 5 shows the results without the data sampling process. Figure 6 shows the optimal results after data sampling. By comparing Figs 5 and 6, it can be found that the model results still improve after sampling the data. RandomForest has the largest PRAUC value improvement of 3.7% after using the sampling method, and GBDT has the optimal PRAUC result of 0.697 in the used model after the sampling process.

Figure 5.

The optimal prediction results were obtained by each machine learning model trained using unsampled data.

Figure 6.

The optimal prediction results were obtained by each machine learning model trained using sampled processed data.

Table 5

The optimal prediction results of each machine learning model are obtained by training with data processed in different data sampling methods

Model	Category	Method	Radio	Accuracy	f1-score	AUC	AUCPRC	MCC	GM
NB	ORG			0.861	0.887	0.864	0.575	0.454	0.659
	Undersampling	NearMiss	20	0.905	0.936	0.855	0.588	0.563	0.649
		RandUnder	80	0.864	0.889	0.865	0.577	0.452	0.663
		ClusterCentroids	80	0.877	0.905	0.863	0.580	0.469	0.661
	Oversampling	BorderSMOTE1	100	0.835	0.852	0.867	0.579	0.431	0.662
		BorderSMOTE2	40	0.838	0.858	0.865	0.579	0.437	0.660
MLP	ORG			0.920	0.968	0.869	0.549	0.572	0.614
	Undersampling	RandUnder	80	0.915	0.959	0.868	0.552	0.575	0.615
		ClusterCentroids	80	0.916	0.965	0.869	0.554	0.567	0.614
		ENN	–	0.907	0.944	0.870	0.555	0.584	0.625
		OSS	–	0.917	0.962	0.870	0.554	0.569	0.610
KNN	ORG			0.928	0.993	0.843	0.525	0.551	0.578
	Undersampling	RandUnder	40	0.926	0.978	0.864	0.549	0.559	0.591
		ClusterCentroids	80	0.927	0.991	0.849	0.526	0.542	0.570
		ENN	–	0.929	0.982	0.853	0.541	0.554	0.581
		RENN	–	0.923	0.969	0.854	0.537	0.552	0.583
		AllKNN	–	0.926	0.978	0.855	0.548	0.565	0.593
		OSS	–	0.930	0.991	0.844	0.532	0.557	0.582
LR	ORG			0.836	0.844	0.864	0.539	0.542	0.593
	Undersampling	ENN	–	0.820	0.823	0.866	0.544	0.559	0.606
		RENN	–	0.803	0.804	0.868	0.568	0.562	0.611
		AllKNN	–	0.818	0.818	0.868	0.553	0.563	0.613
	Oversampling	KMeansSMOTE	100	0.890	0.920	0.864	0.558	0.567	0.610
		SVMSMOTE	40	0.857	0.872	0.863	0.556	0.563	0.608
RF	ORG			0.919	0.965	0.893	0.609	0.601	0.631
	Undersampling	ClusterCentroids	60	0.921	0.965	0.906	0.629	0.598	0.635
		ENN	–	0.910	0.948	0.886	0.617	0.605	0.639
		AllKNN	–	0.908	0.940	0.896	0.613	0.607	0.640
		CNN	–	0.908	0.937	0.888	0.634	0.629	0.659
		OSS	–	0.922	0.966	0.903	0.629	0.614	0.647
	Oversampling	RandOver	20	0.917	0.956	0.901	0.629	0.611	0.646
		SMOTE	60	0.890	0.911	0.906	0.632	0.605	0.641
		KMeansSMOTE	20	0.921	0.962	0.896	0.620	0.598	0.636
		SVMSMOTE	20	0.914	0.954	0.900	0.636	0.613	0.645
		ADASYN	20	0.908	0.940	0.898	0.612	0.611	0.645
	Over+Undersampling	SMOTETomek	60	0.891	0.916	0.900	0.632	0.619	0.657
AdaBoost	ORG			0.925	0.975	0.928	0.609	0.605	0.303
	Undersampling	RandUnder	80	0.924	0.973	0.929	0.623	0.610	0.303
		ClusterCentroids	80	0.918	0.968	0.926	0.611	0.604	0.303
		TomekLinks	–	0.928	0.981	0.931	0.654	0.633	0.303
	Oversampling	RandOver	40	0.922	0.956	0.928	0.620	0.607	0.303
		SMOTE	40	0.911	0.942	0.917	0.616	0.617	0.303
		SVMSMOTE	40	0.917	0.951	0.916	0.623	0.601	0.303
	Over+Undersampling	SMOTETomek	20	0.920	0.964	0.922	0.620	0.601	0.303
XGB	ORG			0.927	0.972	0.921	0.643	0.623	0.661
	Undersampling	ClusterCentroids	80	0.926	0.967	0.922	0.661	0.629	0.667
	Oversampling	SVMSMOTE	20	0.924	0.967	0.917	0.655	0.632	0.667
GBDT	ORG			0.930	0.978	0.940	0.676	0.644	0.684
	Undersampling	ClusterCentroids	80	0.931	0.975	0.939	0.680	0.637	0.674
		OSS	–	0.932	0.974	0.938	0.679	0.635	0.672
	Oversampling	RandOver	40	0.931	0.970	0.942	0.694	0.660	0.696
		KMeansSMOTE	40	0.932	0.971	0.940	0.680	0.640	0.677
		SVMSMOTE	40	0.931	0.965	0.940	0.697	0.654	0.686

To further verify the final results of each machine learning algorithm under different sampling methods and sampling ratios, we collated all the results after sampling, and the final experimental results are shown in Table 5. By analyzing the experimental results in Table 5, we can find that the final results obtained by different models combined with different sampling methods and sampling ratios are different. The experimental results of all eight machine learning experiments were further improved after using the data after data sampling and processing. RandomForest obtained the largest AUCPRC value improvement of 2.7% for the data after sampling with SVMSMOTE. GBDT got the largest AUCPRC value of 0.697 among the eight machine learning prediction models after sampling with SVMSMOTE. GBDT used the RandOver sampling method to make the final combined performance optimal among the eight different machine learning algorithms, the accuracy is 0.931, the AUC is 0.942, and the AUCPRC is 0.694. The results can also provide some reference value for data imbalance experiments combining different algorithms and sampling methods to optimize the experimental results.

4.Discussion

The purpose of this study was to develop a diagnostic model for predicting the severity of patients with COVID-19. Using clinical features and CT radiomics features, the optimal prediction accuracy of 0.932 and AUC value of 0.942 were obtained for the diagnostic model based on the GBDT algorithm after data sampling processing and feature selection. The model can assist clinicians in screening patients with severe COVID-19, providing more medical resources for these patients, and can also be used to improve patient prognosis decisions and assess prognostic treatment outcomes.

Several studies have been conducted to build diagnostic models for the severity of COVID-19 patients using machine learning algorithms, and details of the diagnostic models are shown in Table 6. These studies used patients’ clinical features to build diagnostic models and did not consider the impact of CT radiomics features on model prediction performance. In our study, to further validate the effect of CT radiomics features on the prediction model’s performance, we segmented the ROI on the chest CT by using SP-V-Net and extracted the CT radiomics features on the ROI. Experiments were performed separately using clinical features, CT radiomics features, and a mixture of both. Figure 4 shows that the COVID-19 mild-severe prediction model built by using a mixture of features of both has better performance.

The data imbalance problem was widely present in many real-world decision problems, and in medical diagnosis models, the data imbalance problem can have a negative impact on experimental results. The problem of data imbalance in the studies of Liang et al. [14] and Zhu et al. [15] was not addressed. In our research, to reduce the impact of the data imbalance problem on the prediction results, we sampled the data using three different data sampling methods. By comparing Figs 5 and 6, it can be found that the prediction results of different models were still further improved after the data imbalance treatment. In this study, the optimal prediction results were obtained using integrated models such as GBDT, AdaBoost, and XGB combined with the data after sampling. These classifiers all used integrated learning techniques to improve the accuracy of individual classifiers and overall classifier performance. And related studies have demonstrated that the model prediction performance can be further improved by using integrated models when dealing with data imbalance problems [43].

By further feature ranking and screening of a mixture of CT radiomics features and clinical features, we found that PO2, age, PCT, LDH, and CRP were the five most important clinical risk factors with the most severe degree of prognostic observation for patients with COVID-19, and this result was also consistent with previous related studies. Studies have shown that older age, elevated PCT, LDH, and CRP are all important correlates of the severity of COVID-19 [17]. More important is the continued importance of image features extracted from chest CT to diagnose mild to severe disease in patients with COVID-19. The combination of clinical features and chest CT extracted features has a good performance in diagnosing the severity of COVID-19. COVID-19 patients have specific chest CT image features, including ground-glass opacities (GGO), multifocal patchy consolidation, or interstitial changes in the peripheral distribution [44]. The increase in lesion volume, gross glass volume, and other volumes also provided the possibility for the model to predict the severity of COVID-19.

Machine learning algorithms are now widely used in complementary medical diagnosis and are playing an increasingly important role. Catic et al. [45] built prenatal diagnosis classification models using Artificial Neural Networks (ANNs) artificial neural networks to help physicians in their daily work, eventually obtaining feedforward neural networks with an average accuracy of 89.6% and feedback of 98.8%. Begic et al. [46] applied machine learning algorithms to diagnose congenital heart defects, obtaining a diagnostic accuracy of 94.28% by making the model built by SVM. Stokes et al. [47] applied the trained machine learning model to diagnose and refer to bronchitis and pneumonia. They obtained a 93% AUC value model performance by using decision trees. In our study, the COVID-19 mild-severe diagnostic model built using the GBDT algorithm obtained optimal predictive performance with an accuracy of 93.2% and an AUC of 94.2%. Compared with the above three studies, the model we developed also has good performance in diagnosing mild and severe COVID-19, which can assist physicians in the early detection of severe COVID-19 patients and provide them with better medical resources, and this study has better clinical significance. ML can help improve disease-specific diagnostic systems’ reliability, performance and accuracy. The research and application of ML in the medical field are also increasing, and the related research will provide more convenience for doctors and patients.

The present study still has some limitations that need to be considered. First, the number of patients with COVID-19 is relatively small, limiting the accuracy of the predictive model. Second, the diversity of data in our study is limited, all subjects are Chinese COVID-19 patients, and the results may not be fully applicable to data from other countries. Third, the number of severe patients’ data is small, and the mild and severe patients’ data are imbalanced. We need to collect more COVID-19 patient data, especially severe patients. Further research is still required.

5.Conclusions

This study proves that the COVID-19 mild-severe prediction model based on features extracted from chest CT and clinical characteristics can effectively differentiate the severity of COVID-19 patients and can provide helpful insights for early diagnosis of patients with COVID-19 mild-severe disease. And the prediction models based on both chest CT features and clinical features had higher prediction performance than those built using the two different types of data alone. The results could help clinicians more effectively assess the severity of COVID-19 patients and stratify patients for treatment to reduce potential mortality and ease the burden of care.