Deep learning-based breast region extraction of mammographic images combining pre-processing methods and semantic segmentation supported by Deeplab v3+

2.1.1Traditional pre-processing methods

Pre-processing is an important process in mammographic images, and noise suppression is one of the fundamental operations. Median filter is widely used in medical images. It is effectively a non-linear filter that works well for grayscale images. As well as suppressing speckles and salt and pepper noise, a median filter also preserves the image contours, which is important in this context for being able to adequately distinguish between the pectoral muscle and the breast region. A median filter is a particular kind of low-pass filter. It takes all the pixel values for a designated area within an image and enters them in an element array. The element array is then sorted to arrive at the median value. All the median values for all the element arrays established for all the pixels are then output as an image array. The median filter is applied iteratively until the whole image has been treated. Mammograms have poor levels of contrast and are blurred, which can make them difficult to interpret. Contrast enhancement emphasizes specific regions, thus assisting the next step of segmentation. As in some studies [21, 22, 23], a CLAHE algorithm is applied [24]. It extends upon Adaptive Histogram Equalization (AHE) algorithms by allowing the manual setting of the height of a histogram, which is called the clip limit. The value of the limit is from 0 to 1 and it enables more refined enhancement calculations [21]. Basically, the higher the clip limit, the greater the contrast. Usually, the value for this is set to 0.01 by default. The CLAHE algorithm operates on small pieces of the whole image. The overall image is divided up into blocks called tiles. The contrast transformation function is calculated on each tile so that the contrast is enhanced on each tile. This is based on an approximation that takes both the histogram specified by the distribution value and the specific histogram of the tile into account. The objective is to enhance the contrast for each tile until it more or less matches the overall distribution and desired shape for the histogram. Once the clip limit has been set and the histograms have been redistributed, the transformation function is applied. It works upon a probability density function of the input mammogram’s grayscale value and the total number of pixels in the image. Each of the tiles is combined with one another using bilinear interpolation, which gets rid of any artificially-induced boundaries. The contrast in homogenous areas is limited so as to minimize the risk of amplifying any noise and to reduce any potential shadows at the edges.

Median-filter and CLAHE are widely used in some recent related researches. With regard to the preprocessing of digital mammograms, Lbachir et al. developed a method for handling several of the key steps necessary for the preprocessing of mammograms for CAD systems [25]. They applied a median filter to reduce the salt and pepper noise and a mean filter to deal with false contours produced by the initial noise suppression. Then a CLAHE algorithm was applied. After that artifact removal and label suppression were implemented by applying thresholding and morphological operations. Finally, an active contours method was applied for pectoral muscle suppression. This method was tested on the mini-MIAS database and achieved an accuracy of 98.75%. Basheer et al. [26] have presented a method that first of all flips all the right breast images so that all images are effectively left breast images. Then a thresholding method and an average filter are applied to remove noise and artifacts. For the purposes of pectoral muscle segmentation, a piecewise method is adopted that aims to find the threshold values and find the mid-point between them by comparing and calculating the average values. Finally, translation and rotation are applied to achieve mammogram alignment. The method was tested on the mini-MIAS database and achieved an accuracy of 100% for the removal of noise and labels. The removal of artifacts obtained an accuracy of 99.68%, the pectoral muscle extraction, 92.85%, and the mammogram alignment, 100%. Somewhat similarly, Maitra et al. [21] applied a preprocessing method to the mini-MIAS database that first transformed all images to the same orientation. Then, a CLAHE algorithm was applied for contrast enhancement. Finally, a seeded region growing (SRG) method was applied to achieve image segmentation and identify the pectoral muscles, so that they could be suppressed. This method achieved an accuracy of 95.71%. Adopting a different approach, Bae et al. [27] have proposed a method that focuses on pectoral muscle detection. The artifacts are first removed by applying a labeling algorithm. Then, a median filter is used for noise removal and the CLAHE algorithm is used for contrast enhancement. The detection of candidate pectoral muscle is then achieved by correcting the given image using brightness weights and pectoral muscle edge detection. This results in a candidate contour. The final step is the actual pectoral muscle detection. Here, Hough transformation is used to detect edges and lines within the image. Missing lines are then filled in by connecting lines with similar angles. The pectoral muscle detection is then achieved by applying the RANSAC (random sample consensus) method. This approach produced results with a 92.2% accuracy when applied to the mini-MIAS database.

From the researches above, it can be concluded that noise suppression, artifact and label removal and pectoral muscle suppression are the common ways of implementing pre-processing methods. Through several steps of removing these unnecessary parts, the breast region is naturally extracted. Although traditional methods have reached ideal results such as accuracy, there are still drawbacks. Most of them are time consuming so that majority of the methods cannot be used for real-time operation. Some of the methods (i.e. [21, 26]) have to normalize images by flipping them to face the same direction so as to proceed further operations, which lacks in robustness.

2.1.2Deep learning based semantic segmentation

As the information contained in a mammogram image relates to the breast and pectoral muscle regions, suppressing the pectoral muscle information and enabling segmentation of the regions can be key for the effective use of a mammogram by CAD systems. Clearly, the breast region contains most of the salient detail, so a great deal of research effort has been devoted to effectively bringing the breast information to the fore [23, 29, 30]. Semantic segmentation is usually applied once features are identified. For computer vision systems to accomplish this, it is necessary to undertake classification at the pixel level. This should lead to several groups that each contain all the pixels belonging to one specific category. The Texton Forest method and classifiers based on random forest algorithms were typically used to accomplish this before the widespread adoption of deep learning techniques [31, 32]. After the notable success of Fully Convolutional Network (FCN) models in relation to semantic segmentation [33, 34], there was a move towards using end-to-end convolutional deep learning models instead [35, 36].

Deep learning approaches are increasingly being used throughout medical image processing, not only for mammograms but also for tomosynthesis and ultrasound-based imaging [37]. In the vast majority of cases, some kind of convolutional neural network (CNN) forms the basis of the approach. CNNs are made up of input and output layers with multiple hidden layers in-between that are typically convolutional layers, meaning that they convolve upon the application of some kind of mathematical function. In image processing they can use the convolved layers to learn weights or biases that enable them to assign different degrees of importance to different aspects of an image and to differentiate between them. The first application of a CNN to mammography was by Sahiner et al. in 1996 [38]. Since then, a diverse range of variations upon the basic idea of a CNN have been applied, including a four-layer Adaptive Deconvolution Network (ADM), which was an early kind of CNN used for mass classification [39], the addition of support vector machines (SVMs) for classification [28, 40, 41, 42], modified region proposal CNNs (R-CNNs) [35, 43], combinations of CNN and a stacked autoencoder (CNN-SAE) [44], combining CNN mass detection with extreme learning machine (ELM) clustering [45], the use of a full resolution convolutional network (FrCN) for segmentation before applying a CNN [46, 47], or even a combination of several different kinds of CNNs [48, 49, 50]. To some extent, these variations are the result of what aspect of a mammogram is being focused upon and its perceived utility. Thus, some studies are interested in providing information about variations in breast density [40, 44]. Some want to identify and pinpoint masses [35, 43, 45, 51, 52, 53, 54]. Some are interested in classifying lesions [55]. Some want to facilitate tissue classification [56]. Others want to classify the risk of developing cancer [57, 58] and some are focused on a combination of these various things [59]. One of the key distinctions between various approaches is the approach taken to training the network. In some instances, a pretrained network is used, where the network has already been trained using data from an established database [40, 51, 59], such as ImageNet-1k [60]. Some approaches use fully supervised methods, which explicitly use models to map between different examples and variables, typically to solve classification or regression problems. Others are only semi-supervised [53], making use of a mixture of labelled and unlabeled examples, or weakly supervised [54] making use of data where the labelling is somehow noisy or imprecise. Some studies have adopted an unsupervised strategy, where a model is not used to map to variables but rather to perform operations directly upon the input [44, 45]. These kinds of studies are typically focused on objectives such as clustering or estimating density. At the opposite end of the scale, some networks are hand-crafted, where experts specifically define the features as opposed to them being learned automatically [48, 55].

A number of deep learning approaches have recently begun to be developed that have the capacity to overcome the above issues by taking original images as input, automatically extracting their various features and then learning directly from their individual pixels [28]. They are also associated with improving training precision [61]. A common feature of different deep learning approaches is to convert the original mammograms to a representation of their features across a number of layers, upon which different filters or operations can be applied [28]. Rodriguez-Ruiz et al. [62] have presented a deep learning method that is chiefly focused on pectoral muscle segmentation. The central feature of the method is an automatic model based on a u-net architecture, which is a convolutional network. The u-net architecture is principally an encoder-decoder system. This is highly effective for biomedical image segmentation [63]. The method was tested on 125 images of three different types which are digital breast tomosynthesis (DBT), synthetic mammography (SM) and digital mammography (DM). The results indicated a dice similarity coefficient (DSC) of between 0.947 and 0.970.

There are also some methods developed to solve challenges other than mammogram images. For development of identifying, segmenting and classifying cell membranes and nuclei from human epidermal growth factor receptor-2 (HER2), Saha et al. present a deep learning framework constructed by mainly convolution and deconvolution part [64]. They apply long short-term memory (LSTM) which is a special RNN architecture to help identify and preserve features (i.e. cellular and textural structures) from convolution part then transit to deconvolution part. LSTM is special in avoiding gradient loss or gradient explosion and widely used in speech recognition and handwriting recognition. They use 51 and 28 WSIs from the database of an online HER2 image database of the Department of Computer Science, University of Warwick, United Kingdom, for training and evaluating. The input size of each image is 251*251, They report the effect of the method at precision of 96.64%, recall of 96.79%, f-score of 96.71%, NPV of 93.08%, accuracy of 98.33%, false positive rate (FPR) of 6.84% and AUC of 94%. The authors state that the proposed method is a supplement of research on segmentation and performance measurement of cell membranes and nuclei.

Saha et al. present a deep neural network structured by an encoder, a decoder and a scoring system which is actually a two-stage network [65]. The encoder and decoder part inspired by U-net and Segnet accomplish the goal of segmentation, and the scoring layer is responsible for scoring of ER and PR using IHC images. They use a private database containing 600 images with a size of 2048*1536 which are divided into 100*100 at every batch. A proportion of 65% and 35% for training and evaluating with no overlap existing ensures the number of samples for each process. The batch size of training and evaluating is set to 28, the testing interval is 4500, the maximum iteration is 500,000, the learning rate is 0.01, the weight decay is 0.045 and the momentum is 0.9. The optimizer is stochastic gradient descent (SGD). They achieve the performance of the method at a precision of 95.87%, recall of 95.64%, f-score of 96.49%, NPV of 92.13%, accuracy of 94.53%, AUC of 96% and higher scores than pathologists. The authors claim the proposed method is a valuable supplement of existing ER and PR scoring.

Both of the research above demonstrates the improvement of segmentation using well designed deep learning models, some additional scoring layers give additional evaluation of specific issues. However, the situation of breast extraction in mammogram images is different. Instead of segmenting multiple cells in one image, there is usually one large complete area which needs to extract. This may result in the lack of usefulness of scoring systems which originally work effectively. The need of dividing pectoral muscle from the biggest high gray level areas may demand deeper layers of the model, different levels of features and larger receptive field. Nonetheless, the research above proved the successful apply of deep learning.

One of the goals when using deep learning approaches is to be able to arrive at results that can match, or even exceed human performance, though only a few come near to this at present (i.e. [52]). The accuracy associated with deep learning approaches is variable, ranging from 64% [41], which is only marginally better than conventional CAD-based results to 99% [46, 47]. Semantic segmentation is strongly associated with achieving better results but the use of semantic segmentation is also varied. A notable move towards trying to accentuate the importance of semantic segmentation came with the work of Kisilev et al. [43] where they attempted to apply semantic segmentation in relation to the descriptors commonly used in mammography, i.e. shape, margin and density. Some recent approaches have moved away from conventional CNNs towards other deep learning models, such as conditional generative adversarial networks (cGANs) [66], which can speed up the process by avoiding the need for pixel or patch-based classification, but it is not clear that these achieve any greater degrees of accuracy than the conventional methods. Most recently, the state-of-the-art has moved towards building upon fully convolutional networks (FCNs) by using u-net architectures, using instance segmentation networks and end-to-end convolutional networks, such as DeepLab and Mask RCNN, which promise to offer especially high levels of accuracy [67, 68].

Whilst the capacity of deep learning to improve upon previous approaches to segmentation is clear, there are several problems that have been identified with adopting deep learning approaches that have yet to be fully overcome. One of the main challenges is that the training data is key to the effectiveness of deep learning, so very large training datasets significantly improve the results [61]. However, large public datasets of mammograms are not currently available so most researchers are obliged to still turn to older scanned datasets such as mini-MIAS [37, 69]. This has driven research to using various degrees of supervision [53, 54], transfer learning [59, 70], or the handcrafting of features [48, 52]. Many projects adopt a two-stage process where the first stage involves the detection of candidate lesions before passing potentially malignant results to a deep CNN [52, 69]. Another way around the problem is to avoid a cascaded CNN by using region proposal networks (R-CNNs) [35, 43]. Nonetheless, on the rare occasions that very large proprietary datasets rather than public ones have been made available to certain teams, the results have been far superior [69]. Another noted issue is the degree of complexity and thus greater computational and time-related costs associated with the use of deep learning approaches [46]. With hand-crafting and varying degrees of supervision also being a vital aspect of the training for most methods, there is also an ongoing issue or sourcing the relevant knowledge and the time and cost involved in making it available. Some approaches have also been criticized for being overly-dependent upon subjective criteria such as grayscale intensity [42, 61].

In this study, a DeepLab v3+-based [19] approach is adopted that uses atrous convolution in combination with spatial pyramid pooling and an adapted Xception model [20]. This is a deep convolutional neural network that allows for the capture of rich semantic information and effective semantic segmentation at an adaptable computational cost that can be tailored to real-time needs, while providing higher rates of accuracy than the original DeepLab approach [67, 71]. The principal features of this model are spatial pyramid pooling and an encoder-decoder. Originally, several parallel atrous convolutions (Atrous Spatial Pyramid Pooling) with different rates were used to deliver multi-scale information [71], to good effect. Atrous convolutions (also known as dilated convolutions) introduce a dilation rate that applies a spacing between the values of a kernel so as to expand the field of view at the same basic computational cost. Spatial pyramid pooling enables different layers to use different dilation rates in parallel so as to acquire extremely rich contextual information for segmentation. In [19], this was extended by using Deeplab v3 as an encoder and designing a simple but effective decoder on top. As a further step, a modified Xception model [20] is used as the network backbone, which provides enhanced object detection and semantic segmentation functionality. It thus sets aside many of the potential issues outlined above by delivering semantic segmentation at a lower cost with greater accuracy. This modified network was pretrained on the ImageNet-1k dataset [60], which is based on the WordNet hierarchy, with an average of at least 500 images per node. The approach has previously been tested on the benchmark semantic segmentation dataset PASCAL VOC 2012 [72] and the high-quality annotated dataset Cityscapes [73] and has delivered state-of-the-art results, but it has rarely been applied in the field of mammography.

The main contribution of this paper is shown as follows. The setting of weights for each class help fine-tune the pre-trained Deeplab model which achieved better convergence during training process, so as to accomplish the model with a very small size of mammogram dataset. The input images are processed by proper pre-processing methods, which help improve the performance of the method. Considering the situation of limited samples of mammogram images publicly available, the state-of-the-art performance of the method indicates success of fine-tuning pre-trained networks which is a supplement and improvement of existing methods. The low false positive rate and runtime also strongly recommend a clinical attempt and a fundamental step of further development of CAD systems.

2.2.1Pre-processing

As the mammogram images used to establish this method were from the mini-MIAS database, the original format is pgm and all of the images are in color with the size of 1024*1024. To enable the other steps in the process, they need to be transformed to a jpg format and in grayscale. The process starts with loading each pgm file as a matrix, which is then exported as a jpg file. The jpg file will be judged by the condition of dimension, making sure all files are in grayscale. For the noise suppression step, a median filter with a window size of 3*3 is applied to suppress the noise. Assuming a coordinate (x,y) is in the core of the window, the rest 8 coordinates are (x-1,y-1), (x,y-1), (x+1,y-1), (x-1,y), (x+1,y), (x+1,y+1), (x-1,y+1) and (x,y+1). The new value of (x,y) is the median value of all the 9 coordinates. In order to enhance contrast, a CLAHE algorithm is applied. The tile size was set to 8*8, the clip limit value was set to 0.01, the number of Bins was set to 256, and the distribution of the histogram was uniform. The image is first divided into 8*8 non-overlapping regions, then the histogram is calculated in each region. Each histogram is redistributed in uniform format with a clip limit 0.01, the redundant part will be clipped and averagely inserted to the bottom of the histogram. All histograms are then optimized by a transformation function. The images are resized to 512*512 after contrast enhancement for the next phase. All the process above is implemented in Matlab R2018b. The effect of the pre-processing step is shown in Fig. 2.

Figure 2.

The effects of pre-processing step. The principal noise in the original image (a), is a black line at the bottom of the left-hand image. The image (b) presents the effect of the median filter. The black line has been erased without destroying the other detailed information. The image (c) is the one which applies CLAHE algorithm on image (b). Note the greater visibility of the pectoral muscle in the right-hand image.

2.2.2Semantic segmentation

Semantic segmentation is the novel aspect of the proposed method and is where the advantages of deep learning are brought to bear. Although u-net model has already been applied for breast extraction, Deeplab v3+ model has not yet been applied to solve this challenge which has better performance on semantic segmentation.

The semantic segmentation step applies transfer learning, and a modified xception network is applied as backbone. All the max pooling layers are replaced by depthwise separable convolution layers, and batch normalization and ReLU are added after each 3*3 depth convolution layer. Because the dataset used for training and evaluating is much less than ImageNet, the parameters of the network are not tuned massively. As the number of classes is different between the pretrained network and the training dataset, the last layer is removed remaining logits only. Manual calculated weights are taken into consideration. The value setting of weights of the network is 1.0 by default because ImageNet has a large number of classes, which makes each class containing nearly equal weight. However, the three classes labeled in the training dataset are naturally imbalanced which requires proper value set of weight for each class. The sum of pixels belonging to three classes is calculated, and the least common multiple of three classes is then calculated. The multiple of each class is regarded as weight. The equation of the process can be presented as

where W denotes weight, L denotes least common multiple, C denotes class and n denotes number of the class. The weight of the image is a sum of each class multiplying the related weight. The equation of calculating the weight is

where C denotes the label class, W denotes the weight of the class, i denotes the number of the class from 0 to n, n denotes the total amount of the class. The model takes softmax cross entropy loss as loss function, the equation is shown as follows.

Where X denotes samples, k denotes class, N denotes number of samples of mini-batch, p denotes softmax. The effect of semantic segmentation is shown in Fig. 3.

Figure 3.

Effect of semantic segmentation. Image a and c are input images with different orientations. Image b and d are output labels with breast region extraction as the red part, as well as pectoral muscle region part as the green part.