Construction of U-Net++ pulmonary nodule intelligent analysis model based on feature weighted aggregation

1.Introduction

Lung cancer is a malignant tumor originating from the bronchial mucosa or glands of the lung. Its clinical symptoms are closely related to the tumor size, type, location, stage, complications and metastasis. Early lung cancer patients often have no obvious symptoms, but early detection and treatment have an important clinical significance for prognostic effect. At present, chest computed tomography (CT) is one of the important means in the diagnosis of lung cancer, and pulmonary nodule screening is often used as a key indicator for the early diagnosis [1]. When the pulmonary nodules are presented as single, well-defined, and possess denser soft tissue shadows with a diameter less than 5 mm, they are mostly benign. Pulmonary nodules with spiculation sign at the edges, adhesion to surrounding tissue, and diameter greater than 6 mm are more likely to be malignant.

With the continuous development of medical imaging technology, CT examination is more and more widely used in lung cancer screening. High resolution CT images not only improve the diagnostic accuracy, but also increase the clinical workload of imaging diagnostic doctors. In order to better solve the problem of diagnosis efficiency, and reduce the rate of misdiagnosis and missed diagnosis, computer aided diagnosis (CAD) are employed in the accurate localization and segmentation of pulmonary nodules through imaging diagnostics, image processing technology, and other clinical means. Further, they analyze the tissue components and morphological characteristics of the lesion area and play an active role in the screening of pulmonary nodules [2]. However, considering the influence of the pulmonary nodule size, morphology, pathology and performance of the CT equipment, traditional image segmentation techniques, such as threshold segmentation, region segmentation, edge segmentation and histogram method, can meet the needs of clinical diagnosis when identifying the region information with regular edges, obvious contrast and relative independence. However, in case of presenting the lesion information with blurred boundaries and poor differentiation, there is a greater chance of misdiagnosis or missed diagnosis by using traditional image segmentation techniques.

In recent years, fully convolutional networks (FCN) have gained extensive attention from researchers and have been gradually applied to the practical research of image processing analysis and intelligent diagnosis [3]. Among them, U-Net is currently the most commonly used artificial intelligence image segmentation algorithm, which has the same Encoder-Decoder structure as FCN. U-Net++ is based on U-Net, which adds a dense structure like convolution layer and integrates the features of the next stage of convolution to balance the efficiency and accuracy of the network [4]. Huang et al. proposed the application of U-Net++ model in CT image segmentation to solve the problem of under-segmentation for infiltrating growth [5]. Gui et al. applied the U-Net++ theory to construct myocardial pathology segmentation model, in order to cope with the difficulties, such as small myocardial pathological images and limited sample size [6]. Sui et al. established a segmentation model for the breast region based on U-Net++ and applied it for intelligent MRI image analysis, achieving a good clinical diagnostic effect [7]. In the present study, the CT images of anthropomorphic chest phantom were taken as the research objects, the U-Net++ network structure was used for feature extraction, and the features of different depth levels were weighted and aggregated to explore the intelligent segmentation methods for different types of pulmonary nodules. Our study can provide a reference for improving the accuracy and the processing efficiency of the clinical pulmonary nodules segmentation.

2.Method

Feature enhancement, extraction and classification are the key points in the early screening of pulmonary nodules and the identification of benign or malignant tumors. In this study, following the standardized process of image-aided diagnosis and referring to the relevant theories of image morphological processing technology, the intelligent analysis model of CT images of pulmonary nodules was divided into three stages: image preprocessing, segmentation and auxiliary diagnosis. The specific analysis path is shown in Fig. 1. Firstly, the simulated nodules for experiment were prepared, CT images were collected, training and test samples were screened, and the contrast and sharpness were enhanced. Subsequently, the training samples were manually labeled, and the combination of feature-weighted integration and U-Net++ algorithm was used to segment the pulmonary nodules. The weighted integration U-Net++ model (WI-U-Net++) was constructed and its reliability was evaluated. Finally, the characteristic information of pulmonary nodules was extracted to observe the accuracy of the auxiliary diagnosis in lung cancer.

Figure 1.

Algorithm flow of intelligent analysis model of pulmonary nodules.

2.1Image preprocessing

2.1.1Preparation of simulated nodules

Simulated nodules were prepared by the drip method. Paraffin wax was used as a substrate, and different proportions of magnesium oxide powder were doped to adjust the CT value, so that the CT value was in the soft tissue density domain (20 ∼100 HU). By setting the drip size, drip rate and condensing agent temperature, spheroid nodules with regular edges (type A) and irregular nodules with rough edges and random shape (type B) were prepared. Considering the high diagnostic accuracy of nodules above 8 mm and the low clinical significance of nodules below 2 mm, the conventional size of nodules (2 ∼ 8 mm) was selected in this study.

2.1.2Image enhancement processing

CT images were adjusted to display lung window (WL: -600HU, WW: 1500HU) and DICOM format was converted to TIFF format. The images of the internal pulmonary nodules display layer were included in the study objects and they were numbered according to the type, location and size of nodules. Image enhancement processing included contrast enhancement and edge enhancement. Contrast enhancement adopted Contrast Limited Adaptive Histogram Equalization (CLAHE), which could realize image contrast enhancement, while suppressing the image noise. The edge enhancement adopted the nonlinear edge detector-Kirsch operator, which could find the maximum edge intensity in multiple predetermined directions, that is, different modules were used to carry out the convolution operation on image pixels, and the maximum response value was selected for edge enhancement output [8]. In this study, an 8-module edge enhancement processing model was adopted, as shown in Fig. 2.

Figure 2.

Kirsch operator edge enhancement processing model.

2.2Image segmentation

The purpose of image segmentation is to divide the original image into several specific areas, and classify the areas of interest to assist doctors to diagnose the shape and size of the lesion, which is a commonly used intelligent analysis technology in the field of medical image processing. U-Net++ is an improved model based on U-Net. During training, the output of convolutional layers can be added at different depths to realize the full convolutional network of single input and four outputs. Based on the feature-weighted integration algorithm, the U-Net++ segmentation model for pulmonary nodules was constructed in the present study, as shown in Fig. 3. The U-Net++ network extracted the information of different sampling layers. Subsequently, through training and learning, weighted integration and detailed reconstruction of the layered images was carried out to provide an output of segmented images.

Figure 3.

U-Net++ segmentation model based on feature weighted integration.

2.2.1U-Net++ network framework

Due to the geometric irregularity, random location distribution and large size difference of the pulmonary nodules in CT images, not only the deepest feature information was extracted, but also the superficial feature information was integrated in the process of 4 times down-sampling and 4 times up-sampling of the U-Net++ network. As a result, the learning ability of segmentation model was improved and the segmentation positioning of pulmonary nodules was more accurate. Let x⁢(m,n) define the output of node X⁢(m,n), where m is the number of down-sampling layers along the coding path, and the larger the value is, the deeper features are obtained vertically; n is the order of convolution layers along the skip path, and the larger the value is, the more features of the transverse fusion convolution block. The output calculation method of the convolution block is shown in Eq. (1):

(1)

x⁢(m,n)={H⁢(x⁢(m-1,n)),n=0H⁢([[x⁢(m,k)]k=0n-1,U⁢(x⁢(m+1,n-1))]),n>0

where, H⁢(⋅) is the convolutional block operation, which is used to extract features conducive to the segmentation of pulmonary nodules; U⁢(⋅) is the upper sampling layer, which realizes the feature mapping from a high-dimensional space to low-dimensional space during the segmentation of pulmonary nodules; [ ] is the cascade layer that represents the integration of different feature channels. When n= 0, the node receives information from only the upper layer. When n> 0, the node receives information from the current layer and the lower layer, respectively.

2.2.2Weighted integration

U-Net++ network integrates the feature information of different levels, but these features have different contributions in the prediction label of the model, and the clinical diagnostic information contained in different features is often different. Therefore, the traditional U-Net++ model adopts the average fusion method to limit the further improvement of the accuracy rate of segmentation for pulmonary nodules. Based on the U-Net++ network framework, an adaptive training module was added in this study. When the importance of features at a certain level is higher, the information weight is larger; else, the information weight becomes lower. Based on the different weights, the weighted integration was carried out for the segmentation images with different depths to obtain the weighted segmentation images. In this study, four layers of features were extracted, and the calculation method of image segmentation is shown in Eq. (2):

(2)

W=∑m=1Mαm×wm

where, m is the layer of feature extraction, M is the total number of layers, αm is the weight of the MTH layer segmentation images, and wm is the MTH layer segmentation images. When the model obtains more effective weight through training, the segmentation accuracy gets further improved.

2.2.3Detail reconstruction

In the up and down sampling in U-Net++ network, some details of the anatomical structure and lesion information of CT images can be lost. Therefore, in this study, a detail reconstruction module was added based on feature-weighted integration [9]. The original images and the segmented images of each layer were combined in stages, and the convolution layer of ReLU (Rectified Linear Unit) activation function was adopted to effectively transmit the original image information, eliminate the noise that appeared after the enhanced background area, and obtain the segmented images of pulmonary nodules with high quality details [10].

3.Data acquisition

GE Revolution evo CT was used as image acquisition equipment, AW4.7 was used as image processing workstation and PH-2B chest X-ray phantom was used as the examination objects. According to the anatomical location of the phantom, a total of 60 simulated nodes (30 in class A and 30 in Class B) with different CT values and diameters were uniformly placed in each pulmonary lobe and segment. The phantom was placed in the center of the CT scanning bed and positioned horizontally in the mid-axillary line. The scanning range was from the apex pulmonis to the basis pulmonis. The tube voltages were set at 80 kV, 100 kV, 120 kV and 140 kV, respectively, the Noise Index was NI10∼NI40, and the interval was 2. The reconstruction layer thickness or interval was 0.625 mm/0.625 mm. The reconstruction algorithm was Lung, and Adaptive Statistical Iterative Reconstruction-Veo (ASIR-V) – 40%. A total of 64 sequences and 30,272 images were collected. After screening by AI intelligent analysis platform, 24,800 images that could identify the pulmonary nodules were selected, with type A and Type B accounting for 50% each, which were randomly divided into training set, verification set and test set according to 8:1:1 ratio. The typical AI recognition images are shown in Fig. 4.

Figure 4.

AI recognition of pulmonary nodules in different types of CT images.

4.Results

4.1Segmentation effect of pulmonary nodules

The experimental results of comparison of U-Net algorithm, U-Net++ algorithm and weighted integration U-Net++ algorithm are shown in Fig. 5. Figure 5a–e show the original images, standard images of the simulated nodules, U-Net segmentation images, U-Net++ segmentation images and weighted integrated U-Net++ segmentation images, respectively. The corresponding value of the image is segmentation accuracy (A), which is the ratio (A1/A2) of the overlapping part of the segmented area and the standard area (A1) to the merging part of the segmented area and the standard area (A2). Among them, the standard image of simulated nodule is marked by senior professional radiologists according to the size and characteristics of the built-in nodule.

Figure 5.

Segmentation effect of different algorithms on pulmonary nodules.

As shown in Fig. 5, the WI-U-Net++ algorithm was found to be more conducive to the segmentation for the details and the fuzzy edge structure of pulmonary nodules. Compared with other methods, its segmentation effect was closer to the standard images of the simulated nodules, effectively avoiding the phenomenon of under-segmentation and over-segmentation, and the segmentation accuracy was significantly improved.

Meanwhile, this study analyzed the effectiveness of the algorithm through the segmentation accuracy, false-negative rate and false-positive rate of different types of nodules, and the results are shown in Table 2. The WI-U-Net++ algorithm exhibited a segmentation accuracy of 98.75%, false-positive rate and false-negative rate of 0.00% and 1.25%, respectively, for spheroid nodules (type A), and a segmentation accuracy of 83.47%, false-positive rate and false-negative rate of 5.42% and 11.11%, respectively, for irregular nodules (type B). These results indicate that WI-U-Net++ algorithm could improve the level of intelligent analysis of pulmonary nodules, especially for the irregular nodules with malignant signs, and its clinical application value is more obvious.

5.Conclusion

Lung cancer is one of the most common malignant tumors in clinic, and its CT images show solitary nodules. The texture of benign nodules is relatively uniform, and the grain difference is small, so the images have uniform gray distribution. However the tissue complexity of malignant nodules is high, and the edge structure is irregular. Morphological features are important indicators in imaging diagnosis [12]. With the continuous development of image processing technology, the auxiliary diagnosis software of pulmonary nodules has been applied to a wider extent in the clinic, thereby becoming an important tool in the clinical diagnosis of radiology department. However, the parameters, performance and universality of the software are still different, and the construction of intelligent analysis models for different clinical application purposes has become a research hotspot in this field [13, 14] A major difference in the clinical diagnosis methods and the intelligent analysis model is that the latter only extracts feature information from images, and fails to refer to patient complaints, clinical symptoms, medical history and other information, which is insufficient in analyzing information such as vacuoles and calcification inside nodules. This remains an important research direction to improve the clinical diagnosis value of the machine learning algorithm.

Due to many uncertain factors in the occurrence and development of the pulmonary nodules, CT images can merely display the anatomical structure information within the body layer, but cannot effectively analyze the characteristics of the spatial nodules. In comparison of the segmentation accuracy of the infiltrating pulmonary nodules among U-Net, U-Net++ and other segmentation algorithms in this research field, a significant improvement was observed in the U-Net++ algorithm based on feature weighted aggregation combined with the semantic information corresponding to the features of each layer to improve the definition of the boundary.

In this study, phantom and simulated nodules were used as research objects. The main advantages are as follows: (1) Standard images provide an important reference for the analysis of results, and improve the reliability of the research results. (2) By setting different CT scanning parameters, the phantom can be scanned repeatedly and the obtained image quality is different, which provides enough sample size for the study. (3) The simulated nodules were randomly set according to different shapes, reflecting the diversity of samples and providing sufficient sample size for model training. The results showed that the U-Net++ algorithm based on the feature-weighted integration improved the segmentation effect of pulmonary nodules, especially for irregular nodules with malignant signs. The accuracy of the clinical diagnosis was significantly improved, and the level of diagnosis for benign and malignant differential was improved. We believe that the proposed the U-Net++ algorithm can be an auxiliary diagnostic method with important application value in the intelligent analysis of pulmonary nodules.

In this study, the responses of the auditory system to stimulus of oval and round window were compared with animal experimentation using four Guinea pig. In order to apply the identical vibration stimulus to oval and round window, round window niche was sufficiently drilled out for securing contact area and approaching angle between tip and round window was adjusted with micro-manipulator. After applying the same stimulus with oval and round window, the generated ABR signals were acquired, and the average value of the ABR threshold was compared. And then, the lumped parameter model was proposed that reflects the physical characteristics of the middle-ear and cochlear. In the case of the cochlear, the multi-layered membrane separating scala vestibuli and scala tympani was assumed to be a single membrane. In order to reflect the vibration characteristics of the reverse stimulus, the lumped parameter model was designed in consideration of the helicotrema. The simulated results showed similar to that of animal experiments. The stapes velocity induced to the oval window driving is higher than that in round window driving, and difference of the stapes velocity between oval window and round window increased as frequency increased. As the results of the experiment and simulation, it was confirmed that asymmetrical results were shown according to the stimulus direction. Therefore, the proposed lumped parameter model will be helpful in representing actual biometric characteristics. This model will be useful to understand the cochlear response in round window stimulation.