Classification of fissured tongue images using deep neural networks

2.1Dataset

We collected 3,322 subjects from Heilongjiang University of Chinese Medicine from April 2018 to December 2019. The exclusion criteria for this study were the black race, white race, and subjects with serious mental problems, organic diseases, and cardiac pacemakers. The Institutional Review Boards reviewed the data, and approval was obtained from Heilongjiang University of Chinese Medicine. Written informed consent was obtained from each subject. The overall mean ± standard deviation of the age of male subjects (38%) was 38.86 ± 18.38 years, and that of female subjects (62%) was 39.87 ± 18.55 years.

The dataset was composed of 3322 tongue images measuring either 1480 × 2220 pixels, 1717 × 2579 pixels, or 1640 × 2460 pixels, and the segmentation version of the tongue images was provided. A total of 721 tongue images presented consistent diagnostic results from three TCM practitioners (236 syndrome-related hotness, 306 blood deficiency, and 179 insufficiency of the spleen), whereas the remaining 2601 tongue images had no syndrome results.

2.2Image preprocessing

To maintain the size of the input image, the original pre-segmented tongue image must be scaled. We used the OpenCV toolbox to achieve long-term proportional scaling to 300 × 300 pixels and black pixels to fill any vacancies. In addition, to maintain the continuity of image features during the scaling process and avoid image distortion, the bilinear interpolation algorithm was used in the scaling process.

In this study, we labeled the fissured region using BBox-Label-Tool (https://github.com/NorrisWu/BBox-Label-Tool-master). After identifying the location, the boundary box was saved as (x, y, w, l), where x and y represent the top-left coordinates, and w and l denote the width and height, respectively. Finally, a four-element integer vector was saved to the notepad. Figure 2 shows three samples of tongue images with labels.

Figure 2.

Examples of labeled fissured regions.

2.3Fissured region detection

To learn the features of the fissured region in the tongue image, the region must first be detected. However, the color contrast between the fissured region and its surrounding tissue was unclear, and traditional target detection methods based on hand-designed features often cannot satisfy the requirements. Therefore, at this stage, we used a convolutional neural network (CNN) to determine the fissured region.

We used an SSD in this study, the architecture of which is shown in Fig. 3. The SSD uses a visual geometry group network (VGG16, including 13 convolutional layers and 3 fully connected layers) [20] as the backbone network. The last two fully connected layers were changed to convolutional layers, after which four convolutional layers were added. For an input image, the model first extracts feature maps of different sizes from the backbone network. After another convolutional layer and a pooling layer, every pixel in these feature maps will generate several default boxes of different sizes. Consequently, fissures of different scales can be detected. For every default box, we predicted the category, calculated the classification loss and position loss using the gold standard, and performed training to minimize the loss. Finally, non-maximum suppression was performed to filter overlapping boundary boxes.

Figure 3.

Fissured region detection network structure.

A total of 2601 fissured tongue images were assigned to a training set (70%) or a test set (30%). The training set included 1820 fissured tongue images, and the test set included 781 fissured tongue images. We used stochastic gradient descent to adjust our network to a batch size of 50, a learning rate of 0.0001, a weight decay of 0.0005, and an iteration of 50,000. The non-maximum suppression method was used to filter a part of the overlapping boundary boxes, and we set the overlap rate threshold to 0.6.

During testing, every image was calculated with the fixed weight from the SSD, and the fissured region was output.

2.4Feature extraction

We designed a CNN model named TongueNet to extract both global and local fissured tongue features. These features were used to classify the three syndromes: syndrome-related hotness, blood deficiency, and insufficiency of the spleen.

TongueNet is improved from LeNet [21], which solves the shortcomings of LeNet, i.e., it overfits easily by adding a batch normalization [22] layer and dropout [23]. Furthermore, to ensure that TongueNet learns the features of the fissured tongue image while reducing the training time and parameter calculation, the number of layers was not fixed, and the best model depth was obtained through experimental design.

Figure 4.

Fissure feature extraction model based on G-TongueNet.

The TongueNet architecture used in this study is shown in Fig. 4. TongueNet is primarily composed of several blocks that contain one convolutional layer, one BN layer, and one pooling layer. TongueNet adjusts the depth by increasing or decreasing the number of blocks. Considering the size of the input image, a larger kernel (5 × 5) was used in the first two convolutional layers. As the number of layers increased, we used a smaller kernel (3 × 3). All convolutional layers used the same padding and had one stride. To avoid the disappearance of the gradient, a BN layer was set after each convolutional layer. The ReLU activation function [24] was used in TongueNet because the calculation was rapid regardless of whether forward or backward propagation was involved. All pooling layers were max-pooling layers, with kernels measuring 2 × 2 and two strides. Dropout was used after every pooling layer and fully connected layer. The final Softmax was used to output the probability that the input sample belongs to each class.

TongueNet that extracts local and global features of fissured tongue images is named L-TongueNet and G-TongueNet, respectively. The grid search method was used to determine the best parameters of TongueNet. L-TongueNet achieved the best performance with a learning rate of 0.0001, a weight decay of 0.1, a drop rate of 0.2, a batch size of 60, and a block number of 8. We stopped our training after 500 epochs. Furthermore, G-TongueNet was used to determine the parameters using the grid search method. The main difference between L-TongueNet and G-TongueNet is that the former has 8 blocks, but the latter has 7 blocks. In addition, InceptionV3 [25, 26] and ResNet18 [27] were trained for comparison.

During testing, the network served as a fixed feature extractor. We eliminated the last layer of Softmax and output the pre-layer as deep features; therefore, we extracted global features from the original fissured tongue image and local features from the fissured region detected by the SSD. All these features were used to classify the three syndromes. Although the network can output the classification directly, we discovered that the fusion features were better. Comparisons and analyses are shown in Section 3.

2.5Classification of syndromes

At this stage, the global and local features were concatenated in the channel dimension. Subsequently, the GBDT was trained to classify the three syndromes. The GBDT builds multiple decision trees sequentially, where each decision tree is built on the residual from the previous decision tree.

Two classic classifiers, the SVM and back propagation (BP) neural networks [28, 29], were used for model comparison. The experimental results were evaluated using the following six metrics: (1) accuracy; (2) sensitivity; (3) specificity; (4) time complexity; (5) space complexity; and (6) number of parameters [30].

To eliminate the effects of accidental factors on the accuracy of the model, bootstrap and permutation tests were performed. All confidence intervals were computed based on the percentiles of 1,000 random resamplings (bootstraps) of the data. The accuracy difference and p-values were calculated by a standard permutation test [31] using 10,000 random resamplings of the test set. Firstly, the test set was sampled repeatedly to obtain two sample groups, which were respectively input to model A and model B. Subsequently, the accuracy difference between model A and model B was calculated; repeating 10,000 times, obtains 10,000 differences. Finally, we obtained the average accuracy difference. The confidence interval and p-value were also calculated from the 10,000 differences.

3.1Results of fissured region detection

A total of 733 fissured regions were correctly detected, and the precision, recall, and mAP reached 0.81, 0.88, and 0.85, respectively. Figure 5 shows some fissured regions detected by the SSD in the test set. The results indicate that: 1) the SSD performance was good, i.e., 88% of fissures with all shapes were detected; and 2) some small fissures were overlooked. This was because small fissures, typically with shallow depths, have a similar contrast to surrounding tissue and hence are difficult to detect.

Figure 5.

Some fissured regions detected by SSD.

3.2Global and local features

We conducted the experiments on our dataset using two additional illustrious methods, i.e., ResNet18 and InceptionV3. Using the training strategy described in feature extraction, we trained L-ResNet18 and L-InceptionV3 from the fissured regions and G-ResNet18 and G-InceptionV3 from the whole tongue images.

Figure 6.

ROC of the global feature models.

To illustrate the results more clearly, we constructed a receiver operating characteristics (ROC) curve of each syndrome, as shown in Fig. 6. Furthermore, we calculated the accuracy, sensitivity, specificity, and area under the curve (AUC) values, as shown in Table 1. Although G-TongueNet indicated the highest AUC value in the classification of syndrome-related hotness, ResNet18 attained the maximum AUC value in the classification of blood deficiency and insufficiency of the spleen. Different models attained their maximum values under different syndromes and metrics.

Figure 7.

ROC of local feature models.

Table 2 shows the metrics of different global feature learning models. To avoid the contingency of the test set, we used bootstrap to resample the test set 1000 times and calculated the 95% confidence interval. The accuracy difference, representing the average difference in accuracy between two models in 10,000 permutation tests, and the p-value were also calculated. Firstly, the accuracy difference was valid and a p-value less than 0.005 signifies that G-TongueNet was better than G-ResNet18 and G-InceptionV3. Meanwhile, G-TongueNet performed better in terms of the number of parameters, time complexity, and space complexity. It can be concluded that G-TongueNet can significantly reduce the training time and model storage cost while obtaining the best deep features.

Similar to the global feature extraction experiment, we constructed the ROC curve and created a metric table from the local feature extraction experiment, as shown in Fig. 7 and Table 3. The results show that L-TongueNet, L-ResNet18, and L-InceptionV3 yielded similar performance, and no particular method triumphed in all metrics.

Furthermore, the number of parameters, space complexity, time complexity, and overall accuracy are shown in Table 4. Similarly, L-TongueNet performed the best in terms of number of parameters, space complexity, and time complexity. Although L-TongueNet performed better than L-InceptionV3, it exhibited the same overall accuracy as L-ResNet18. The results indicate that L-TongueNet and L-ResNet18 possessed the same feature extraction capabilities, which were better than those of L-InceptionV3. Additionally, L-InceptionV3 trained 415 times more parameters than L-TongueNet and exhibited worse accuracy due to overfitting.

3.3Classification results

From the experiments in the last part, it is evident that G-TongueNet and L-TongueNet performed the best in terms of global and local feature extraction; therefore, we used the 128-dimensional deep feature at the last fully connected layer of the G-TongueNet and L-TongueNet as the output. In this experiment, 721 tongue images were segmented into a training set and a test set at a ratio of 7:3. The training and test sets contained 505 and 216 images, respectively. Based on the model structure described in Section 2, after the fusion of global and local features, we used the GBDT to classify the syndromes. In addition, the SVM and BP networks were used for comparison.

Using the grid search method to adjust the parameters, we discovered that the linear kernel SVM with cost C = 15 performed the best. The GBDT with learning_rate = 0.2, n_estimators = 40, subsample = 0.8, min_samples_leaf = 80, and min_samples_split = 100 yielded the highest accuracy in the training set. In addition, the BP network with a learning rate of 0.001, a weight decay of 0.9, a batch of 60, and a number of hidden layer nodes of 140 performed the best in the training set.

Figure 8.

ROC of different classifiers.

Figure 8 shows the ROC curve of each syndrome, and Table 5 shows the accuracy, sensitivity, specificity, and AUC. In terms of syndrome-related hotness and blood deficiency, the three curves were relatively close to each other; however, in terms of insufficiency of the spleen, the three curves differed, wherein the SVM performed better. Meanwhile, Table 5 shows the performance of the three methods more clearly. We discovered that in syndrome-related hotness classification, the GDBT achieved the highest accuracy, specificity, and AUC values; however, the SVM achieved the best accuracy, sensitivity, and AUC values. In blood deficiency classification, the GBDT achieved the highest accuracy and sensitivity values, but the BP network achieved the highest values for all metrics. Finally, in terms of insufficiency of the spleen, the SVM achieved the highest accuracy, specificity, and AUC values, whereas the three methods exhibited the same sensitivity value. It is noteworthy that, for some metrics, different classifiers achieved the highest value simultaneously; therefore, we believe that the difference among the three classifiers was minimal.

Furthermore, we calculated the overall accuracy of the three classifiers, as shown in Table 6. The GBDT exhibited minimal advantages; however, from the results of the permutation test, we discovered no significant difference in the model accuracy (p-value > 0.05). Based on Table 6, we further prove that no significant difference existed among the three classifiers.

In addition, the classification results of G-TongueNet and L-TongueNet were compared with those of the fusion feature model using the GDBT. The results are shown in Table 7. The accuracy of the model using fusion features was 3% higher than that of G-TongueNet and 5% higher than that of L-TongueNet. After 1000 bootstrap replications, the average accuracy difference was calculated. Moreover, our fusion feature model still surpassed G-TongueNet and L-TongueNet by 2.5% and 5.3%, respectively. We believe that our method, which combines global and local features, can improve the accuracy of classification in syndromes.