Comparative analysis of weka-based classification algorithms on medical diagnosis datasets

1.Introduction

With the advent of 5G and the era of Big Data, the rapid development of medical information technology around the world, the massive application of electronic medical records and cases, and the digitization of medical equipment and instruments, a large amount of data has accumulated in the database system of hospitals, which includes clinical diagnosis data and hospital management data. The data sources are mainly electronic medical record systems, examination systems, testing systems, ultrasound systems, physical examination systems, and office automation systems. The analysis and mining of these data can gradually put together the information in the “fragmented” data accumulated in traditional medicine so as to present a more comprehensive picture of life conception and disease development. This can help people to have a more comprehensive and in-depth understanding of the mechanism of life and the principles of disease generation, thus improving the efficiency of disease prevention and control and clinical treatment.

With the continuous development of cross-cutting disciplines, a large number of machine learning and artificial intelligence algorithms are applied to medical datasets as an important component in disease prediction models, demonstrating excellent performance in medical-related fields such as disease prediction and assisted diagnosis, drug selection and application, and health insurance fraud and detection [1, 2, 3, 4]. As important artificial intelligence techniques such as data mining and machine learning, the decision tree algorithms in Weka are often used in basic medical research areas such as classification of DNA genes, classification and comparison of DNA barcodes, development and implementation of siRNA design tools, screening and differentiation of salt-loving and non-salt-loving proteins, and differentiation of classification properties of cell death-related proteins [5, 6, 7, 8, 9]. The main applications in the biomedical field are in the prediction, diagnosis, and treatment of diseases. J48 algorithm has achieved good results in diagnosing neonatal xanthogranuloma [10] and predicting the incidence of stroke using the k-nearest neighbor and C4.5 decision tree [11]. Also, scholars have used the multilayer perceptual decision tree algorithm to mine the decision of breast cancer treatment to assist doctors in the diagnosis and selection of hormone therapy, radiotherapy, chemotherapy, and other treatments [12], evaluation of the feasibility of support vector machine (SVM)-based diffusion tensor imaging for the classification of childhood sexual epilepsy disorders [13], and classification studies based on electroencephalogram (EEG) features of Alzheimer’s disease biomarkers [14], among others. In the area of hospital operations management services, the J48 algorithm has also been applied to machine learning related to electronic health record typing [15]. Stiglic [16] used supervised learning to select small datasets from medical error data that could be effectively predicted to enable the prediction of medical errors with a small set of datasets. SVMs were used to predict and design DrugMint, a web server for drug molecules [17]. Kaijian et al. [18] mined and analyzed chronic kidney disease data on the basis of the Weka data mining platform, and found that Random Forest (RF) performed better in the classification of chronic kidney disease datasets through algorithm comparison analysis. Ying et al. [19] used Weka software to build machine learning models. The results showed six algorithms with better classification prediction on the diabetes dataset: logistic model tree (LMT), sequential minimal optimization (SMO), Logistic, Naive Bayes, Rotation Forest, and Bagging. Yadin [20] employed the ID3 algorithm using data mining techniques to analyze dental consultation data. The ID3 algorithm was effectively improved, and the improved algorithm was again applied to the data. The accuracy greatly improved, and the researcher achieved the expected decision tree and classification rules.

Previous studies mostly applied single or several classifiers of the same type for modeling. The drawback of this approach was that algorithms that were truly applicable to a certain domain might be missed, and comprehensive modeling using algorithms based on different ideas was rarely used. In this study, we aimed to further explore the application of machine learning techniques in different medical fields by applying machine learning techniques to different medical prediction datasets using Weka data mining software, constructing multiple classification models, and analyzing the models through multiple performance evaluation metrics, so as to better inform hospital-oriented clinical and management applications.

2.Concepts and methods

Weka, a publicly available data mining workbench, assembles a large number of machine learning algorithms capable of undertaking data mining tasks, including pre-processing data, classification, regression, clustering, association rules, and visualization on a new interactive interface. In this study, we selected 24 machine learning algorithms in 6 major categories, which were highly representative (Table 1).

The simple Bayesian algorithm is a commonly used probabilistic classification algorithm having the advantages of fast computation, high accuracy, and operational simplicity. In practical applications, it is assumed that the attributes in the samples affect them independently of each other. Therefore, it is productive for most of the more complex problems. Algorithms such as multilayer perceptron, logistic regression, and neural networks are function-based classification algorithms, and some of these algorithms are based on kernel functions and some on decision functions. Locally weighted learning (LWL) is an important algorithm in lazy learning. When a new instance needs to be processed, the distance between the training instance and the test instance is calculated using a distance function to determine a weighted set of training instances associated with the test instance, which is then used to construct a new model to process the new instance. The Euclidean distance is usually used to measure the distance between instances in traditional weighted learning algorithms. The Bagging algorithm, which was proposed by Breiman, was one of the first and the simplest integration method with the best performance. The basic principle is mainly to use a weak classification algorithm and an original training set. The learning algorithm is used to train classifiers in multiple rounds. Each round of training requires a bootstrap procedure to randomly have put-back sampling from the original training set to reconstitute a new training set for training a base classifier. After the training is completed, a sequence of prediction functions can be obtained, and eventually the individual prediction functions are classified using the simple voting method for the samples to be tested. Each random sampling obtains a different subset of training samples; hence, different base classifiers are trained, thus ensuring the diversity of the base classifier. The partition and regression tree (PART) decision tree algorithm, an algorithm invented in 1998 by Eibe Frank and Ian H. Witten, is an algorithm that uses incomplete decision trees to extract rules in a dataset. RF, J48, and other algorithms are based on decision trees. RF involves bootstrap aggregation in integration learning. The basic idea is to train a set of identical base classifiers (decision trees) and integrate the predictions based on all base classifiers using the integration methods such as hard voting, weighted voting, and so forth to obtain the final prediction results.

3.Experiment

4.Analysis of experimental results

4.2Intra-group comparison analysis of classification algorithms

This subsection conducted experimental validation for each of the three classes of algorithms to further verify the effectiveness and robustness of algorithms based on different ideas on different medical classification datasets. Among these, five metrics such as MAE, RMSE, RAE, RRSE, and FPR were negative metrics whose smaller values indicated better classification of the algorithms; the larger values of the remaining metrics indicated better classification of the algorithms.

4.2.1Meta-based algorithm

Ensemble learning is the combination of several weak classifiers (which can also be regressors) to produce a new classifier. As shown in Fig. 1, the Bagging algorithm outperformed the other algorithms on all datasets by 11 metrics, mainly on the 2 dichotomous datasets. Logit Boost, as a direct optimization of the binomial logarithmic loss Adaboost algorithm, had only seven metrics that performed significantly. Dagging provided a subset of the classified training data to the chosen base learning algorithm. The best algorithm was Rotation Forest, with a total of 28 indicators achieving optimal values, because the diversity of trees in the Rotation Forest was obtained by training each tree in the rotated feature space of the entire dataset before running the tree induction algorithm. Rotating the data axes created completely different classification trees. Besides ensuring tree diversity, the rotated trees reduced the constraints on the univariate trees that could decompose the input space to a hyperplane parallel to the original feature axes. More specifically, the complete feature set was created for each tree in the forest using a feature extraction approach. Principal component analysis (PCA) is a common unsupervised learning method that uses orthogonal transformations to transform observations represented by linearly correlated variables into data represented by several linearly uncorrelated variables. Each element was a linear combination of the original data. Further, the first major element was guaranteed to have the maximum variance. The other elements had high variance under the condition that they were orthogonal to the original elements, which showed that the RF algorithm was suitable for large data volume and high-dimensional linear classification data.

Figure 1.

Classification effect of the algorithm based on the integration idea.

Figure 2.

Classification effect of the algorithm based on tree theory.

4.2.2Algorithm based on tree theory

For tree-thinking classifiers, the LMT algorithm achieved optimal results on all metrics on the MAGR dataset, which showed that the distance-based function classification was better under the small-sample low-dimensional linear condition as shown Fig. 2. BFTree, J48, and Random Tree performed poorly on each dataset. The best algorithm was RF, which achieved optimal results on the ILPD, CADG, and LYMP datasets with a total of 27 metrics reaching the optimum. In terms of the correct classification rate of the algorithms, all algorithms achieved more than 90% correct classification rate on LYMP, and the RF algorithm even exceeded 95%. The main idea of RF was to select a feature at each node of the tree from a subset of different attributes and replace the more general “random subspace approach” [21], which could be applied to many other algorithms such as SVMs. However, a recent comparison between RFs and random subspaces for decision trees has shown that the former is superior to the latter in terms of accuracy [22]. Other ways of adding randomness to the decision tree induction algorithm exist especially when numerical characteristics are involved. For example, instead of using all instances to decide the best splitting point for each numerical characteristic and using subsamples of instances [23] that are different. The optimal splitting criterion was evaluated using these characteristics and splitting points, which were chosen by each node decision. Since the sample selection for splitting at each node was different, the result of this technique was an ensemble of different tree combinations.

Table 5

Accuracy of various algorithms

Class or classification	Algorithm	MAGR	ILPD	CADG	LYMP
		Acc (%)	Acc (%)	Acc (%)	Acc (%)
Bayes	Naive Bayes	78.46	67.33	70.93	92.81
	BayesNet	82.83	70.87	73.71	95.86
Functions	SMO	79.29	69.13	78.46	94.12
	RBF Network	78.04	67.87	81.46	92.81
	Logistic	82.31	70.8	82.5	93.25
	Multilayer Perceptron	81.69	72.27	82.93	94.77
Lazy	LWL	81.89	66.47	45.44	82.79
	KStar	81.27	87.53	80.1	96.51
	IBk	75.55	75.47	78.65	95.42
Meta	Bagging	83.14	81.07	85.04	93.25
	Logit Boost	83.14	72.93	84.81	94.34
	Dagging	81.37	67.73	71.5	93.46
	Rotation Forest	82.93	82.27	86.59	95.86
Rules	Decision Table	82.31	68.33	67.73	90.85
	PART	81.79	73.8	83.44	93.46
Tree	BFTree	82	78.2	81.98	93.46
	J48	82	77.2	83.4	93.9
	LMT	84.39	76.47	85.23	94.99
	Random Forest	79.19	87.4	87.91	95.86
	Random Tree	76.48	78.53	78.97	91.94

5.Conclusions

In this study, we analyzed and evaluated the applicability and classification ability of each class of algorithms by comparing the effectiveness of machine learning algorithms on medical datasets with different patterns. Machine learning models can assist doctors to make predictions based on diagnosis and treatment information. However, in actual medical scenarios, doctors need to combine multiple aspects of patient information and their own empirical knowledge to comprehensively assess the disease. At the same time, they may encounter data category imbalance problems, multimodal data fusion problems, and knowledge reuse during classification and prediction. In the future, we can analyze and manage the patient consultation data by relying on our 360 panoramic electronic medical record system and Internet hospital platform. Also, we can develop more accurate prediction models, better integrate machine learning technology into the diagnosis and treatment analysis process, and truly play the role of assisting doctors through the introduction of classifier accuracy performance adjustment tools and the addition of more disease-specific test data.