Advertisement

Automatic segmentation of glioblastoma multiform brain tumor in MRI images: Using Deeplabv3+ with pre-trained Resnet18 weights

      Highlights

      • The Deep-Net sensitivity in delineation of the enhanced tumor (ET) is more than 90%.
      • Dice Similarity Coefficient above 67% was achieved for all tumor regions.
      • The obtained DSC for ET is 78%, which is among the highest scores in the literature.
      • Combination of Deeplabv3+ and Resnet18 weights can perform well in segmenting the GBM tumor sub-regions.

      Abstract

      Purpose

      To assess the effectiveness of deep learning algorithms in automated segmentation of magnetic resonance brain images for determining the enhanced tumor, the peri-tumoral edema, the necrotic/ non-enhancing tumor, and Normal tissue volumes.

      Methods and Materials

      A new deep neural network algorithm, Deep-Net, was developed for semantic segmentation of the glioblastoma tumors in MR images, using the Deeplabv3+ architecture, and the pre-trained Resnet18 initial weights. The MR image Dataset used for training the network was taken from the BraTS 2020 training set, with the ground truth labels for different tumor subregions manually drawn by a group of expert neuroradiologists. In this work, two multi-modal MRI scans, i.e., T1ce and FLAIR of 293 patients with high-grade glioma (HGG), were used for deep network training (Deep-Net). The performance of the network was assessed for different hyper-parameters, to obtain the optimum set of parameters. The similarity scores were used for the evaluation of the optimized network.

      Results

      According to the results of this study, epoch #37 is the optimum epoch giving the best global accuracy (97.53%), and loss function (0.14). The Deep-Net sensitivity in the delineation of the enhanced tumor is more than 90%.

      Conclusions

      The results indicate that the Deep-Net was able to segment GBM tumors with high accuracy.

      Abbreviations:

      FLAIR (Fluid Attenuation Inversion Recovery), CNN (Convolutional Neural Network), NCR\ECT (Necrotic and Non-Enhancing Tumor), HGG (High-Grade Glioma), ET (Enhancing Tumor), PE (Peri-tumoral Edema), GT (Ground Truth), LGG (Low-Grade Gliomas), 2D (Two-dimensional), MICO (Multiplicative Intrinsic Component Optimization), ReLU (Rectified Linear Unit), BN (Batch Normalization), NT (Normal Tissue), SGD (Stochastic Gradient Descent), BS (Batch Size), ILR (Initial Learning Rate), LRDF (Learn Rate Drop Factor), L2R (L2 Regularization), JSC (Jaccard Similarity Coefficient), IoU (Intersection Over Union), TP (True-Positive), TN (True-Negative), FP (False-Positive), FN (False-Negative), TC (Tumor Core), WT (Whole Tumor), BFScore (Boundary F1 Score)

      Keywords

      Introduction

      One of the challenging issues in medical practice is the determination of the tumor location and its extension, i.e., tumor segmentation. Tumor segmentation is an essential step in evaluating the tumor situation and its treatment response follow-up [
      • Pirtoli L.
      • Gravina G.L.
      • Giordano A.
      Radiobiology of Glioblastoma: Recent Advances and Related Pathobiology.
      ]. Currently, the most common approach in the segmentation of the tumors is manual tumor delineation, which is a time-consuming process, which needs the long work of a group of clinical expert neuro-radiologists. Therefore, it can be easily affected by human error. Nowadays, the need for accurate and automatic diagnosis and treatment methods in health care is increasing, because of the increasing number of patients due to the population growth, and the rising demand for better, and faster health services. Recently, computer-aided techniques [
      • Zikic D.
      • Ioannou Y.
      • Brown M.
      • Criminisi A.
      Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
      ,
      • Vaidhya K.
      • Thirunavukkarasu S.
      • Alex V.
      • Krishnamurthi G.
      Multi-modal brain tumor segmentation using stacked denoising autoencoders.
      ,
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.-M.
      A convolutional neural network approach to brain tumor segmentation.
      ,
      • Randhawa R.S.
      • Modi A.
      • Jain P.
      • Warier P.
      Improving boundary classification for brain tumor segmentation and longitudinal disease progression.
      ,
      • Shaikh M.
      • Anand G.
      • Acharya G.
      • Amrutkar A.
      • Alex V.
      • Krishnamurthi G.
      Brain tumor segmentation using dense fully convolutional neural network.
      ,
      • Kim G.
      Brain tumor segmentation using deep fully convolutional neural networks.
      ,

      Moreno Lopez M, Ventura J. Dilated convolutions for brain tumor segmentation in MRI scans. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:253–62. 10.1007/978-3-319-75238-9_22.

      ,
      • Papadimitroulas P.
      • Brocki L.
      • Christopher Chung N.
      • Marchadour W.
      • Vermet F.
      • Gaubert L.
      • et al.
      Artificial intelligence: Deep learning in oncological radiomics and challenges of interpretability and data harmonization.
      ,
      • Castiglioni I.
      • Rundo L.
      • Codari M.
      • Di Leo G.
      • Salvatore C.
      • Interlenghi M.
      • et al.
      AI applications to medical images: From machine learning to deep learning.
      ] have increased the accuracy and speed of the tumor segmentation process with the potential of reproducibility in medical applications and reducing the human dependency on the process. There are many challenging factors in the segmentation of complicated tumors like Glioblastoma Multiform, GBM, such as localization, accurate border delineation, and precise determination of tumor extension area [
      • Nadeem M.W.
      • Al Ghamdi M.A.
      • Hussain M.
      • Khan M.A.
      • Khan K.M.
      • Almotiri S.H.
      • et al.
      Brain tumor analysis empowered with deep learning: A review, taxonomy, and future challenges.
      ,
      • Dehkordi A.N.V.
      • Kamali-Asl A.
      • Ewing J.R.
      • Wen N.
      • Chetty I.J.
      • Bagher-Ebadian H.
      An adaptive model for rapid and direct estimation of extravascular extracellular space in dynamic contrast enhanced MRI studies.
      ]. GBM are usually diffused tumors with ambiguous boundaries and very different shapes and sizes. Magnetic Resonance Imaging (MRI) is an effective imaging technique to analyze brain GBM tumors. Different sequences of MRI can help to distinguish the different regions of the tumors [

      Shukla G, Alexander GS, Bakas S, Nikam R, Talekar K, Joshua D. Advanced magnetic resonance imaging in glioblastoma : A review Advanced magnetic resonance imaging in glioblastoma : a review. Chin Clin Oncol 2017:40. 10.21037/cco.2017.06.28.

      ,
      • Zegers C.M.L.
      • Posch J.
      • Traverso A.
      • Eekers D.
      • Postma A.A.
      • Backes W.
      • et al.
      Current applications of deep-learning in neuro-oncological MRI.
      ,
      • Dehkordi A.N.V.
      • Kamali-Asl A.
      • Wen N.
      • Mikkelsen T.
      • Chetty I.J.
      • Bagher-Ebadian H.
      DCE-MRI prediction of survival time for patients with glioblastoma multiforme: using an adaptive neuro-fuzzy-based model and nested model selection technique.
      ]. The enhancing tumor region is more clearly displayed in T1-weighted or in contrast-enhanced T1-weighted (T1ce) MR images [

      Shukla G, Alexander GS, Bakas S, Nikam R, Talekar K, Joshua D. Advanced magnetic resonance imaging in glioblastoma : A review Advanced magnetic resonance imaging in glioblastoma : a review. Chin Clin Oncol 2017:40. 10.21037/cco.2017.06.28.

      ]. Whereas, the edema and necrotic area are better highlighted in the Fluid Attenuation Inversion Recovery (FLAIR) images [

      Shukla G, Alexander GS, Bakas S, Nikam R, Talekar K, Joshua D. Advanced magnetic resonance imaging in glioblastoma : A review Advanced magnetic resonance imaging in glioblastoma : a review. Chin Clin Oncol 2017:40. 10.21037/cco.2017.06.28.

      ]. Therefore, the combination of the different MRI sequences in automatic brain tumor segmentation can be very useful in providing complementary information about the whole tumor and improving the segmentation performance.
      Deep neural networks are capable of automatically extracting the appropriate features from the image and then learning and recognizing the pattern of the tumors and normal tissues, so the deep-learning algorithms have been widely used for tumor segmentation in the last few years [
      • Rüfenacht E.
      • Jungo A.
      • Ermiş E.
      • Blatti-Moreno M.
      • Hemmatazad H.
      • Aebersold D.M.
      • et al.
      Organs at Risk Delineation for Brain Tumor Radiation Planning in Patients with Glioblastoma Using Deep Learning.
      ,

      Inmaculada M, Ocaña G, Román KL, Urzelai NL, Ángel M, Ballester G, et al. Deep Learning in Healthcare. vol. 171. 2020.

      ,

      Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

      ]. Convolutional Neural Network (CNN) structure is the most popular structure among the deep neural networks for segmentation tasks [
      • Zikic D.
      • Ioannou Y.
      • Brown M.
      • Criminisi A.
      Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
      ,
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.-M.
      A convolutional neural network approach to brain tumor segmentation.
      ,
      • Nadeem M.W.
      • Al Ghamdi M.A.
      • Hussain M.
      • Khan M.A.
      • Khan K.M.
      • Almotiri S.H.
      • et al.
      Brain tumor analysis empowered with deep learning: A review, taxonomy, and future challenges.
      ,

      Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

      ,
      • Mazzara G.P.
      • Velthuizen R.P.
      • Pearlman J.L.
      • Greenberg H.M.
      • Wagner H.
      Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation.
      ]. So far, the different dimensional CNN with 2D [
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.-M.
      A convolutional neural network approach to brain tumor segmentation.
      ,
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      ], 2.5D [
      • Wang G.
      • Li W.
      • Ourselin S.
      • Vercauteren T.
      Automatic Brain Tumor Segmentation Based on Cascaded Convolutional Neural Networks With Uncertainty Estimation.
      ], or 3D structures, [
      • Ahmad P.
      • Qamar S.
      • Hashemi S.R.
      • Shen L.
      Hybrid Labels for Brain Tumor Segmentation.
      ,
      • Kamnitsas K.
      • Ledig C.
      • Newcombe V.F.J.
      • Simpson J.P.
      • Kane A.D.
      • Menon D.K.
      • et al.
      Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.
      ,
      • Isensee F.
      • Kickingereder P.
      • Wick W.
      • Bendszus M.
      • Maier-Hein K.H.
      No new-net.
      ,
      • Myronenko A.
      3D MRI brain tumor segmentation using autoencoder regularization.
      ] were used in tumor segmentation. 2D CNNs (slice-by-slice form) are extensively used in designing the deep-learning-based algorithm due to their less memory requirement. Using higher-dimensional CNN helps to preserve more image information at cost of more memory and time required. 2.5D CNN is used as a trade-off between inter-slice features and memory consumption [
      • Wang G.
      • Li W.
      • Ourselin S.
      • Vercauteren T.
      Automatic Brain Tumor Segmentation Based on Cascaded Convolutional Neural Networks With Uncertainty Estimation.
      ].
      The feature extraction capability of the network has a vital role in the CNN outcome. In deep CNNs, the convolutional layers are responsible for the automatic extraction of the features. Various structure designs [

      Moreno Lopez M, Ventura J. Dilated convolutions for brain tumor segmentation in MRI scans. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:253–62. 10.1007/978-3-319-75238-9_22.

      ,
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      ,
      • Havaei M.
      • Davy A.
      • Warde-Farley D.
      • Biard A.
      • Courville A.
      • Bengio Y.
      • et al.
      Brain tumor segmentation with Deep Neural Networks.
      ,
      • Cui S.
      • Mao L.
      • Jiang J.
      • Liu C.
      • Xiong S.
      Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
      ] and different feature extraction techniques such as the combination of the local and global feature extraction[
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.
      A Convolutional Neural Network Approach to Brain Tumor Segmentation.
      ], using a secondary training stage that trains the network by patches uniformly from the image[
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.-M.
      A convolutional neural network approach to brain tumor segmentation.
      ] and fusing the 2D images in axial, sagittal, and coronal views to make 3D features and provide more information for the network [
      • McKinley R.
      • Wepfer R.
      • Gundersen T.
      • Wagner F.
      • Chan A.
      • Wiest R.
      • et al.
      Nabla-net: A deep dag-like convolutional architecture for biomedical image segmentation.
      ,
      • Li Y.
      • Shen L.
      Deep learning based multimodal brain tumor diagnosis.
      ,
      • Hu X.
      • Li H.
      • Zhao Y.
      • Dong C.
      • Menze B.H.
      • Piraud M.
      Hierarchical multi-class segmentation of glioma images using networks with multi-level activation function.
      ] or different weight balance functions [
      • Brosch T.
      • Tang L.Y.W.
      • Yoo Y.
      • Li D.K.B.
      • Traboulsee A.
      • Tam R.
      Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation.
      ,
      • Ronneberger O.
      • Fischer P.
      • U-net T.B.
      Convolutional networks for biomedical image segmentation.
      ] were employed by the previous researches to enhance the feature extraction capability of the CNNs. A group of investigators [
      • Zikic D.
      • Ioannou Y.
      • Brown M.
      • Criminisi A.
      Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
      ,
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      ,
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.
      A Convolutional Neural Network Approach to Brain Tumor Segmentation.
      ,
      • Menze B.H.
      • Jakab A.
      • Bauer S.
      • Kalpathy-cramer J.
      • Farahani K.
      • Kirby J.
      • et al.
      ] tried to optimize the previously reported architectures or developed structures with less computing need to make the process faster or to prevent the network from overfitting.
      Pre-trained deep neural networks are also widely used in previous investigations for tumor segmentation. Reasonable results reported in the published works encouraged the investigators to develop their deep-learning segmentation algorithm based on the pre-trained networks [
      • Kim G.
      Brain tumor segmentation using deep fully convolutional neural networks.
      ,

      Moreno Lopez M, Ventura J. Dilated convolutions for brain tumor segmentation in MRI scans. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:253–62. 10.1007/978-3-319-75238-9_22.

      ,
      • Ahmad P.
      • Qamar S.
      • Hashemi S.R.
      • Shen L.
      Hybrid Labels for Brain Tumor Segmentation.
      ,

      Pawar K, Chen Z, Shah NJ, Egan G. Residual encoder and convolutional decoder neural network for glioma segmentation. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:263–73. 10.1007/978-3-319-75238-9_23.

      ,
      • Kermi A.
      • Mahmoudi I.
      • Khadir M.T.
      Deep convolutional neural networks using U-Net for automatic brain tumor segmentation in multimodal MRI volumes.
      ,
      • Choudhury A.R.
      • Vanguri R.
      • Jambawalikar S.R.
      • Kumar P.
      Segmentation of brain tumors using DeepLabv3+.
      ,
      • Marcinkiewicz M.
      • Nalepa J.
      • Lorenzo P.R.
      • Dudzik W.
      • Mrukwa G.
      Segmenting brain tumors from MRI using cascaded multi-modal U-Nets.
      ]. The Deeplabv3+, introduced by Chen et al [
      • Chen L.-C.
      • Zhu Y.
      • Papandreou G.
      • Schroff F.
      • Adam H.
      Encoder-decoder with atrous separable convolution for semantic image segmentation.
      ] is an encoder-decoder architecture and is especially well adapted to the prediction of pixel labeling. In this work, a deep learning-based algorithm (Deep-Net) was developed and well optimized for semantic segmentation of the glioblastoma tumor in MR images, using the Deeplabv3+ architecture, and the pre-trained Resnet18 initial weights. To improve the segmentation performance of the tumor sub-regions, the combination of two MRI modalities including T1ce and FLAIR images were employed to train the deep network and segment the whole brain images into four classes, i.e., The Gd-enhancing tumor, the peri-tumoral edema, the necrotic and non-enhancing tumor (NCR\ECT), and Normal tissue. Since, deep learning models are strongly dependent on the model hyper-parameters, such as Batch Size, Initial Learning Rate, Learn Rate Drop Factor, Epoch number, L2 Regularization, etc.; finding the optimized values for these parameters in such a high-dimensional space is a serious challenge, and can significantly affect the behavior of the training process, the performance of the trained model, and also the cost function value. This paper addresses the need for obtaining an optimized package of the model hyper-parameters, so far lacking in the scientific literature. Therefore, this work has focused on this issue by executing a lot of models with a wide investigating range for each hyper-parameter.

      Materials and methods

      Datasets

      The Dataset used in the experiment was taken from the BraTS2020 training set [
      • Menze B.H.
      • Jakab A.
      • Bauer S.
      • Kalpathy-cramer J.
      • Farahani K.
      • Kirby J.
      • et al.
      ,
      • Bakas S.
      • Akbari H.
      • Sotiras A.
      • Bilello M.
      • Rozycki M.
      Data Descriptor : Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features.
      ,
      • Bakas S.
      • Reyes M.
      • Jakab A.
      • Bauer S.
      • Rempfler M.
      • Crimi A.
      • et al.
      Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge.
      ]. The BraTS2020 dataset included sample multi-institutional routine clinically-acquired pre-operative multimodal MRI scans (T1-weighted, T1(ce)-weighted, T2-weighted and FLAIR) of 369 patients—293 glioblastoma/high-grade gliomas (GBM/HGG) and 76 low-grade gliomas (LGG), with accompanying ground truth labels for different tumor sub-regions manually drawn by a group of expert neuro-radiologists. The ground truth, GT, annotations comprise the GD-enhancing tumor (ET), the peri-tumoral edema (PE), and the necrotic and non-enhancing tumor core (NCR/NET). All 3D-MR images of the BraTS’2020 dataset have the dimension of 240 × 240 with 155 slices through the whole brain and skull stripped. In this work, two multi-modal MRI scans, i.e., T1ce and FLAIR of 293 patients with high-grade glioma (HGG), were used for deep learning network training (Deep-Net).

      Data preparation and image preprocessing

      Since the developed Deep-Net used two-dimensional (2D) CNNs, first, all 3D data were converted into 2D slices from the axial view, and 90,830 2D slices of two MRI modalities were prepared. The slices with tumor information less than 0.5% of the whole brain volume were removed from the dataset to reduce the memory requirement and to improve the network performance [
      • Randhawa R.S.
      • Modi A.
      • Jain P.
      • Warier P.
      Improving boundary classification for brain tumor segmentation and longitudinal disease progression.
      ]. Thus, 35,008 slices were picked up to form the dataset of this study. Secondly, labeled images were created according to the ground truth labels which were recognized by the group of radiologists to identify the different parts and distribution of the glioblastoma tumor. All regions out of the tumor and inside the brain were considered normal tissues. Therefore, as it can be seen in Fig. 1, each labeled image consists of five classes, i.e., normal tissue (blue), peritumoral edema (yellow), NCR\ECT (gray), the Gd-enhancing tumor (red), and background (black).
      Figure thumbnail gr1
      Fig. 1The perpetration and preprocessing steps applied to the images before use as input images of Deep-Net.
      A few image processing steps were applied to the images before recruiting them in the training of the developed deep network. As the intensity value of the MR image depends on the imaging protocol and scanner used, the intensities in the original MR images were normalized to a value between 0 and 255. Tissue inhomogeneity may lead to significant misclassification in the segmentation procedure [
      • Li C.
      • Gore J.C.
      • Davatzikos C.
      Multiplicative intrinsic component optimization (MICO) for MRI bias fi eld estimation and tissue segmentation.
      ,
      • Gallivanone F.
      • D’Ambrosio D.
      • Carne I.
      • D’Arcangelo M.
      • Montagna P.
      • Giroletti E.
      • et al.
      A tri-modal tissue-equivalent anthropomorphic phantom for PET, CT and multi-parametric MRI radiomics.
      ], therefore, A multiplicative intrinsic component optimization (MICO) [
      • Li C.
      • Gore J.C.
      • Davatzikos C.
      Multiplicative intrinsic component optimization (MICO) for MRI bias fi eld estimation and tissue segmentation.
      ], was also applied to the images to correct the inhomogeneity through a specific tissue and prevent the overlay between the ranges of the intensities of different tissues. In the next step, to limit the image to brain borders, the extra background parts of the original images with the dimension of 240 × 240 were cropped and then reduced to 224 × 224 using the nearest-neighbor interpolation technique, because the input of our developed deep neural network, Deep-Net, needs an image input with the size of 224 × 224 × 3, where 3 is RGB channels. Since the used data is grayscale, each image was repeated three times to fill the three channels. Fig. 1 illustrates the preparation and preprocessing steps applied to the images before using them as input images of Deep-Net.

      Computing platform

      This study was conducted on the MATLAB 2021a platform [

      MATLAB 2021a, The MathWorks Inc. Deep Learning Toolbox™. Natick, Massachusetts, United States; 2021.

      ] using the Deep Learning Toolbox. The GPU, NVIDIA GeForce RTX 3080 with 12 GB RAM, in addition to CPU Intel Core i7-10700 K (@3.5 GHz) with 32 GB RAM were used on the Windows 10 system.

      2–3-The Deep-Net architecture

      The developed Deep-Net in this study is a pixel-wise segmentation network based on Deeplabv3+ architecture and the pre-trained Resnet18 initial weights [
      • Chen L.-C.
      • Zhu Y.
      • Papandreou G.
      • Schroff F.
      • Adam H.
      Encoder-decoder with atrous separable convolution for semantic image segmentation.
      ]. The Deeplabv3 + is based on an encoder-decoder architecture as shown in Fig. 2.
      Figure thumbnail gr2
      Fig. 2The architecture of the developed Deep-Net and its components based on the Deeplabv3 + structure.
      Encoder: A pre-trained CNN extracts basic information from the input image for segmentation tasks [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Spatial Pyramid Pooling in Deep Convolutional.
      ]. The Encoder component is composed of convolution layers with a max-pooling layer and dilated convolutional layer, based on Atrous convolutions, the Atrous Spatial Pyramid Pooling (ASPP) module. The baseline CNN consists of 4 convolution layers with Rectified Linear Unit (ReLU) activation function and a kernel size of 7 × 7 in the first layer and 3 × 3 in others. After each convolution layer, there is a batch normalization layer (BN). After the first convolution layer, a 3 × 3 max-pooling layer was employed that shrinks the size of the feature map. The size reduction (downsampling factor) is controlled by the pooling size and the value, which corresponds to the horizontal and vertical increments at the positioning of pooling sub-windows. The produced feature map imports the ASPP module with rates 6, 12, and 18 (for three parallel 3 × 3 convolution branches with a default downsample rate of 16) to extract the multi-scale contextual information of images by creating a larger field of view [
      • Choudhury A.R.
      • Vanguri R.
      • Jambawalikar S.R.
      • Kumar P.
      Segmentation of brain tumors using DeepLabv3+.
      ]. This pathway leads to extracting the image context and informative features map from the input images but in reduced size and higher depth. The output stride value in an encoder part can be very effective in the accuracy and computation speed of the network, two possible stride values (8 and 16) are available in the Deeplabv3 + structure.
      Decoder: The information extracted from the encoding step is used to create an output image similar to the size of the input image. The decoder block includes bilinear Up-sampling layers (deconvolution layers with stride 4), concatenation (concatenation layer takes inputs and concatenates them along a specified dimension), the SoftMax layer, and the classification layers. The decoder components recover the size of the feature map to the size of the input image and add the spatial resolution of the features to the map, which was lost during the downsampling. Since, low-level features have more spatial information and inclusion with high-level features enhances the segmentation performance, to preserve the intermediate low-level features produced in encoder layers, a 1 × 1 convolution is performed on the feature maps produced in the intermediate output of the encoder, and then this contextual information is concatenated with the up-sampled encoded features by an 8 × 8 convolution with stride 4. A 3 × 3 convolution filter is also applied and then an up-sampling process by a factor of 4 was employed again to produce the final segmentation output [
      • Chen L.-C.
      • Zhu Y.
      • Papandreou G.
      • Schroff F.
      • Adam H.
      Encoder-decoder with atrous separable convolution for semantic image segmentation.
      ]. Lastly, a SoftMax non-linearity layer was employed that normalizes and assigns a multi-nominal distribution over the different labels of classes, and then, the pixels of each image were classified into 5 classes.
      This structure needs a feature extraction network in its encoder part. A few of pre-trained networks (such as Resnet18 [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ], Resnet50 [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ], Resnet101 [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ], mobilenetv2 [
      • Sandler M.
      • Howard A.
      • Zhu M.
      • Zhmoginov A.
      • Chen L.C.
      MobileNetV2: Inverted Residuals and Linear Bottlenecks.
      ], Xception [
      • Xception C.F.
      Deep learning with depthwise separable convolutions.
      ] and Efficientnet [

      Tan M, Le Q V. EfficientNet: Rethinking model scaling for convolutional neural networks. 36th Int Conf Mach Learn ICML 2019 2019;2019-June:10691–700.

      ]) are capable to employ as extractor at encoder part of Deeplabv3+. Among the possible networks, the Resnet18 weights as its initial weights were recruited in our algorithm. A combination of Deeplabv3+ and Resnet18 weights were adapted and optimized according to our data. Resnet18 is an 18-layer network and has trained on more than one million images from the ImageNet database [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ]. This pre-trained network needs for calculation of 11.7 million parameters to be performed. The input images of Resnet18 are 224 × 224 × 3. To get the pre-trained Resnet18, first, Deep Learning Toolbox™ Model for the Resnet18 network in MATLAB2021a software was installed. Then utilizing the “deeplabv3plusLayers” command, the layers and weights of Resnet18 were added to the Deeplabv3+ structure.

      Training of the developed Deep-Net

      Among the prepared 2D slices for all 293 patients, the 26256 slices of 220 patients were used as the training set, 1750 slices of 15 patients were employed as a validation set, and the 7002 slices of the remained 58 patients were recruited as a testing set. Four main classes were considered to evaluate the Deep-Net efficiency: enhanced tumor (ET), NCR\ECT, peri-tumoral edema (PE), and normal tissue (NT).
      One of the main challenges during the training process is the volume of each class or the imbalance between the number of pixels. This may cause a defect in the segmentation, because training is dominated by the classes with more pixels, like non-tumor pixels, and normal classes, while the algorithm goal focuses on the classes with the fewer pixels, like ET. To overcome this issue, different balance techniques such as median frequency balancing technique [
      • Bischke B.
      • Helber P.
      • Borth D.
      • Dengel A.
      Segmentation of imbalanced classes in satellite imagery using adaptive uncertainty weighted class loss.
      ], inverse frequency balancing technique [
      • Tian J.
      • Mithun N.
      • Seymour Z.
      • Chiu H.-P.
      • Kira Z.
      Striking the Right Balance.
      ], and uniform class balancing technique [

      Ayhan B, Kwan C. Tree, shrub, and grass classification using only RGB images. Remote Sens 2020;12. 10.3390/RS12081333.

      ] were employed to find the best balancing approach to adjust the class volume, and improve the efficacy of the Deep-Net for the smaller classes.
      In this study, the cross-entropy loss function [
      • Zhang Z.
      Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels.
      ] was employed to guide the learning process to calculate the error between the Deep-Net output and the labeled ground truth, and the stochastic gradient descent (SGD) algorithm was utilized as an optimizer to update the weights and reach the minimum lost.

      Deep-Net hyper-parameters optimization

      Adjustment of the network hyper-parameters can significantly affect the network performance. Each hyper-parameter can directly change the weight of network layers and also affect the high features learning capability, which leads to a different convergence point [
      • Zhao M.
      • Xin J.
      • Wang Z.
      • Wang X.
      • Wang Z.
      Interpretable Model Based on Pyramid Scene Parsing Features for Brain Tumor MRI Image Segmentation.
      ]. Therefore, the following network parameters were investigated to find their optimal values and maximum Deep-Net performance.

      Batch size (BS)

      Since batch size controls the number of training samples in each iteration, it can have a great effect on the accuracy of the trained network. The advantage of using small batch size values is its less memory requirement, and fast training process. However, increasing the possibility of the noise effect, and also network divergence, is the disadvantage of small BS [
      • Shirish Keskar N.
      • Mudigere D.
      • Nocedal J.
      • Smelyanskiy M.
      • Tang P.T.P.
      On large-batch training for deep learning: Generalization gap and sharp minima.
      ]. Very large batch sizes are usually accompanied by a drop in the generalization ability of the network [
      • Shirish Keskar N.
      • Mudigere D.
      • Nocedal J.
      • Smelyanskiy M.
      • Tang P.T.P.
      On large-batch training for deep learning: Generalization gap and sharp minima.
      ]. A good investigation range of the batch size in deep neural networks is 8 to 128. Therefore, the batch sizes of 8, 16, 32, 64, and 128 were used in this study to find the optimum value.

      Initial learning rate (ILR)

      The ILR determines the learning speed of the training model and also affects the accuracy of the trained Deep-Net. A few different values in the range of 0.001 to 0.02 were checked and the ILR corresponding with the best performance without sacrificing the speed of the training was chosen as the optimum value. As a too high ILR may cause to lose of the local minima of the lost function, we limited our trial-and-error range of ILR to less than 0.02, besides the very low ILR may either take a long time to converge or get tramp in undesirable local minima, so the investigated range was considered more than 0.001.

      Learn rate drop factor (LRDF)

      The LRDF introduces the updated learning rate in each step by multiplying the initial learning rate by a determined drop factor. This parameter can be tuned to a value between 0 and 1. A few various drop factors in the range of 0.1 to 0.8 were assessed. The learning rate drop periods of 5 and 10 epochs were investigated in this study.

      Epoch

      The epoch number determines the number of passes of the entire dataset to the network. Most deep neural networks usually reach a relatively saturated accuracy after a specific epoch, after which the accuracy improvement is not significant. The optimal epoch was obtained as the epoch located at the 10% of the plateau region of the validation accuracy curve.
      The shuffle function was also employed to riffle the data for each epoch to have different data for each batch. The data shuffling can help to decrease variance and keep the model general, and lessen the over-fit.

      L2 regularization (L2R)

      L2R was used to reduce the network complexity and avoid network overfitting. A few common values such as 0.0001, 0.005, and 0.01 were investigated to reach the best loss function for our developed Deep-Net. Of course, in this study, an extensive investigation for finding the optimal regularization rate was not performed.

      Deep-Net evaluation with performance metrics

      To quantify the performance of the trained Deep-Net, 20% of all the dataset was selected as test data, and a few metrics including global and mean accuracy, mean and weighted Intersection over Union, IoU, and Boundary F1 Score, BFScore, were calculated to assess the global performance of the Deep-Net in segmentation. The Sørensen-Dice Similarity Coefficient, DSC [
      • Shamir R.R.
      • Duchin Y.
      • Kim J.
      • Sapiro G.
      • Harel N.
      C ontinuous Dice Coefficient : a M ethod for Evaluating P robabilistic S egmentation s.
      ], and Intersection Over Union (IoU), also known as the Jaccard similarity coefficient (JSC), were computed to evaluate the similarity percentage between the Deep-Net predicted labels with the ground truth labels for each 5 considered classes. The tumor sub-regions were also grouped into three classes and the DSC and JSC were computed to assess the three constructed regions: enhanced tumor (ET), Tumor Core or TC (including the Enhanced tumor region with NCR\ECT), and Whole Tumor or WT (including the Tumor core and Edema region).
      The performance evaluation metrics [
      • Csurka G.
      • Larlus D.
      • Perronnin F.
      • Meylan F.
      What is a good evaluation measure for semantic segmentation?.
      ] used in this study are as follows:
      • (1)
        Accuracy or global accuracy= (TP + TN)/ (TP + TN + FP + FN)
      • (2)
        Sensitivity, Recall, or true positive rate TPR = TP/(TP + FN)
      • (3)
        Specificity, selectivity, or true negative rate, TNR = TN/(TN + FP)
      • (4)
        Precision = TP/ (TP + FP)
      • (5)
        BF score = 2 * precision * recall / (recall + precision)
      • (6)
        Dice Similarity Coefficient (DSC) = 2TP/(2TP + FN + FP)
      • (7)
        Intersection over Union, or JSC = TP / (TP + FP + FN) = (Pred GT)/ (Pred GT)
      Where TP is True-Positive, TN is True-Negative, FP is False-Positive, FN is False-negative, Pred is the predicted map, and the GT is the ground truth.

      Results

      Deep-Net optimization results

      Fig. 3A shows the correct classification fraction, CCF, or the global accuracy of the optimal Deep-Net at different epochs respectively. The CCF increases by the epoch number until the curve reaches a plateau, and does not increase any further at higher epochs. The epoch number corresponding to 10% of the plateau region of the graph was taken as the stopping or optimum epoch. As shown in the figure, the 37th epoch is chosen as the optimal epoch, or the termination epoch for training the Deep-Net. The validation accuracy of the optimized Deep-Net at optimum epoch was achieved at 97.5%. In the other words, there is a 2.5% chance of failure for the trained Deep-Net to provide a correct classification when it is applied. Fig. 3B shows the variation of the cross-entropy loss function as a function of the epochs number. The loss function at the optimal epoch is 0.14. As shown in the figures, the improvement of the accuracy and loss values in greater epochs are not significant.
      Figure thumbnail gr3
      Fig. 3A: correct classification fraction (CCF), or global accuracy of the trained Deep-Net at different epoch numbers. B: calculated loss function in validation process of the Deep-net.
      The developed Deep-Net was optimized for a few hyper-parameters. Figs. 4A to 4D illustrate the variation of the loss function and CCF with different values of the aforementioned hyper-parameters. Fig. 4A indicates that the small batch sizes are insufficient to train the network and improve the global accuracy. However, a further increase in the batch size from 64, gives rise to performance reduction and the batch size investigation was end up at 128. Based on the results shown in the Figure, the BS of 64 with the best network accuracy and loss was picked up as the optimum batch size. Fig. 4B displays the dependence of the accuracy and loss function on the ILR. As shown in the figure, the optimal ILR with minimum loss and maximum accuracy was obtained at ILR = 0.009. LRD factor was also inspected for a few values in two different drop periods. A reduction factor in the range of 0.1 to 0.8 was applied after every 10 epochs and the correct classification factor of the Deep-Net was calculated at each factor. The solid curve in Figure-4C represents the Deep-Net global accuracy for tested LRDF with a drop period of 10 and the dashed curve illustrates the Deep-Net loss function values versus various LRDFs at this drop period. As can be seen in the figure, the performance outperformed when the LRDF was equal to 0.4. To further investigate the Deep-Net performance, the optimized value of the LRDF, 0.4, was once tested on the network with a drop period of 5 and the results were compared with the drop period of 10. According to the results, the network, with a drop period of 5 gives an accuracy of 96.98%, and a loss of 0.165, which is not better than the drop period of 10 (accuracy = 97.5%, and loss = 0.14). Therefore, the combination of LRDF = 0.4, and a drop period = 10, was more adequate and selected as the optimal set of hyper-parameters. Fig. 4D shows the evaluation of Deep-Net performance against different regularization rates. As shown in the figure, the regularization rate of 0.005 among the three evaluated regularization rates corresponds to better performance and was employed in the training process of Deep-Net. the results also implied that the L2 regularization rate of 0.0001 has signs of network overfitting.
      Figure thumbnail gr4
      Fig. 4The percentage of the Deep-Net accuracy and cross-entropy loss values versus A: investigated batch sizes B: different ILRs, C: a wide range of LRDFs, and D: different L2 regularization rates.
      The algorithm was also tested for two down sampling rates of 16 and 8. The down sampling rate significantly affects the processing time, a smaller down sample, results in a larger feature maps dimension (28 × 28), and a longer up-sampling process to recover the size of the image. The results indicate that different down sampling rates can influence the divergence process and consequently the algorithm efficacy. The results showed that the algorithm with a down sample rate of 8 reached a better global accuracy (97.5%) than the down sample rate of 16 (97.13%) with a cost of the slower training process (4 h and 30 min in return of 3 h and 30 min). This accuracy improvement impresses a 2–3% increment in the prediction of the whole tumor region and also a 2% increment on the enhanced tumor area. As the sampling rate becomes smaller, the number of weights that are applied to the valid feature region becomes larger, which makes the computing time of the algorithm to be longer. Using the down sample rate of 8 also makes the ASPP module rates to be double (i.e., rates of 12, 24, and 36). To speed up the training process (∼1.30 times faster) and reduce the memory consumption, a higher down sampling rate was used and tested. Down sample of 16 reduces the training times, however, this may come at the cost of missed small objects and reduced accuracy at semantic boundaries. The calculated evaluation scores for a trained network with a down sample of 16 (validation accuracy 97.13%, validation loss 0.146) are as follows:
      • Global accuracy (or CCF) calculated in confusion matrix for 5 classes: ET = 89.04%, PE = 95.7%, NCR\ECT = 87.37%, NT = 96.8% and BG = 99.55%
      • DSC, Mean IoU (JSC), precision, sensitivity and BF score are 80%, 73.87%, 79%, 95% and 69.21%, respectively for PE class, 65.75%, 73.47%, 83%, 87% and 79% respectively for NCR\NET class, 77.41%, 76.51%, 84%, 89% and 84.42% respectively for ET class and 98.18%, 96.52%, 99%, 97% and 88.41% respectively for NT class.
      • DSC and JSC are 85.50%, 84.56%, 77.41% and 76.61%, 78.06%, 67.34% respectively for WT, TC and ET classes. The results of the trained network under down sample rate of 16 are also promising on different evaluation metrics with a higher speed in the training process.

      Performance metrics results of the optimized Deep-Net in class segmentation

      After choosing the best hyper-parameters, the trained Deep-Net was evaluated by 7002 test data. The mean overall performance of the developed semantic segmentation algorithm was evaluated with different performance metrics. The global and mean values of accuracy, mean IoU, weighted IoU, and mean BF score were found to be 97.5%, 94%, 85%, 95%, and 87% respectively.
      Table 1 tabulates the confusion matrix of algorithm performance for 5 classes: peri-tumoral edema (PE) as class 1, NCR\ECT as class 2, Enhanced tumor (ET) as class 3, Normal tissue (NT) as class 4, and background (BG) as class 5. The confusion matrix of the Deep-Net gives a lot of information about the network performance. The confusion matrix revealed that the constructed Deep-Net shows high performance in classifying the five aforementioned classes correctly. The Deep-Net efficacy for all classes is above 87%. As shown in the Table 1, the diagonal elements of the CCF confusion matrix (sensitivity) for the five output classes were found to be 95.43%, 87.35%, 90.53%, 97.35%, and 99.44% for Classes 1 to 5 respectively. The sensitivity, specificity, and Type I error of Deep-Net can be easily calculated from the confusion matrix. The specificity was calculated to be 93.1%, 92.98%, 88.45%, 96.45%, and 99.11% for classes 1 to 5 respectively, and Type I error, false positive, were found to be 6.9%, 7.02%, 11.55%, 3.55%, and 0.09% for the five classes. However, the non-uniform distribution of the pixels into five different classes may cause the segmentation network to become biased towards the classes with larger volumes and increase the Type I error in smaller classes, but the problem was relatively solved using the median frequency balancing technique.
      Table 1A 5 × 5 normalized the confusion matrix of correctly detected pixels for each class by the trained Deep-Net.
      ClassesPE (predicted)NCR\ECT (predicted)ET (predicted)NT (predicted)BG (predicted)
      Scores
      PE (actual)95.410.671.592.320.02
      NCR\ECT (actual)2.6187.359.780.250.01
      ET (actual)2.686.3190.530.450.02
      NT (actual)1.60.040.1797.350.84
      BG (actual)0.010.0040.0140.5399.44
      * PE: Peri-tumoral edema, NCR\ECT: Necrotic and Non-Enhancing tumor, ET: Enhanced tumor, NT: Normal tissue, BG: Background.
      Then, to better analyze the segmentation power of the Deep-Net to classify the brain pixels into different classes, different scores (DSC or F1 score, mean BF score, Mean IoU or JSC, precision and sensitivity or recall) were used for network performance evaluation for the four main classes, which are listed in Table 2. The scores were not reported for the 5th class, or BG, due to its less importance. All these evaluation scores were measured with the same test data. The normal tissue (NT) has a larger volume between all classes and the trained Deep-Net for segmentation of the normal pixels just encounters a challenge in the pixels around the tumor or the pixels in the boundary of the brain. Therefore, specificity and type I error are very small for this class and as can be seen in Table 2, all calculated scores for this class are above the 97%. This table implies that the tumor region and its structure can be reliably segmented by the trained Deep-Net. DSC and JSC indicate the similarity of the segmented maps by Deep-Net with ground-truth maps. DSC above 67% was achieved for all tumor regions. An important class to be accurately segmented is enhanced tumor (ET). The attained sensitivity for ET is 90%. The Dice similarity coefficient for ET is 78%, which is among the highest obtained DSC score in the literature [
      • Zikic D.
      • Ioannou Y.
      • Brown M.
      • Criminisi A.
      Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
      ,
      • Vaidhya K.
      • Thirunavukkarasu S.
      • Alex V.
      • Krishnamurthi G.
      Multi-modal brain tumor segmentation using stacked denoising autoencoders.
      ,
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      ]. The high sensitivity (high accuracy) and high precision (high reliably reproducible) for all classes reveal that the trained Deep-Net is accurate and reliable enough to be used in clinical segmentation practice. The BF score measures how close the predicted boundary of an object matches the ground truth boundary and is defined as the harmonic mean (F1-measure) of the precision and sensitivity values with a distance error tolerance (the default value is 0.75% of the length of the image diagonal). The value was calculated for each class and tabulated in Table 2.
      Table 2The evaluation scores to measure the Deep-Net efficacy for classes 1 to 4.
      ClassesPENCR\NETETNT
      Scores
      DSC (F1 score)81%67%78%98%
      Mean IoU (JSC)77%75%78%97%
      Precision79%83%84%99%
      Sensitivity (Recall)95%87%90%97%
      Mean BF score76%81%87%91%
      * PE: Peri-tumoral edema, NCR\ECT: Necrotic and Non-Enhancing tumor, ET: Enhanced tumor, NT: Normal tissue.
      To compare the results with the results of previous investigations, the tumor region was grouped into three classes and all evaluation scores were measured for the new three classes too: Class A = Whole Tumor (WT), Class B = Tumor Core (TC), Class C = Enhancing Tumor (ET). The classification of the pixels into the three mentioned classes was performed on the predicted segmentation label maps and ground truth label maps, and a 0 and 1 mask was prepared for each class. Then, the two prepared maps were compared. The DSC of 87%, 85%, and 78% and JSC of 79%, 79%, and 68% for Class A, Class B, and Class C reveal that the segmented maps by the Deep-Net are strongly in agreement with the ground truth maps.

      Visualize evaluation of the Deep-Net performance

      The eight exemplary segmentation result of the developed and trained algorithm with the respective ground truth images is included in Fig. 5. Four sample slices were chosen from T1ce MR images and the other four slices were selected from the FLAIR MR images. The first column illustrates the bias-corrected MR images and the second column displays the provided ground truth maps. As shown in the first column, the extra background of the brain was cropped in each image to minimize the frequency difference between the inside brain and the background. The segmented regions by the trained Deep-Net are shown in the third column. The Deep-Net predicted maps were overlapped on the ground truth maps and the difference between the two maps is illustrated in column 4. The colors in the images of this column indicate the success or non-success performance of the trained network. The black, light, and dark gray shows the correct segmented pixels by the Deep-Net, the green pixels illustrate the pixels that the Deep-Net was not able to detect, and the purple color pixels are the pixels that are wrongly segmented by the Deep-Net. The last column of Fig. 5, the dice score, quantitate the similarity between the segmented regions by the Deep-Net and the GT maps. The exemplary slices in this figure visually imply that the trained Deep-Net successfully classifies the brain image pixels into the four main classes for both T1ce and FLAIR images.
      Figure thumbnail gr5
      Fig. 5Comparison of the segmented tumor regions by the trained Deep-Net and the ground truth maps for eight exemplary slices from the test data are shown in this figure. The first four slices are T1ce MR images and the last four slices are FLAIR MR images. The last column displays the measured dice similarity index for the eight sample slices.
      However, the Deep-Net can perform well for most tested slices, it has a few defects for some tested slices to accurately segment the pixels of the enhanced tumor, especially for the thin and small areas of the enhanced tumor. Fig. 6 demonstrates the six example slices to show some caused defects (specified by the black Arrows) by the Deep-Net in the segmentation of the classes. As can be observed in this figure, another weakness of the Deep-Net for some slices is the integration of pixels around a class as a part of that class.
      Figure thumbnail gr6
      Fig. 6Six example slices from six different high-grade glioma patients to show Deep-Net defects in the classes segmentation.

      Ablation study results

      The median frequency balancing technique was applied in all tested models. To evaluate the importance of this balancing technique to the achieved results, the median frequency balancing was replaced by inverse frequency balancing and uniform class balancing techniques in the optimized model. The calculated Dice scores for ET, TC, and WT using inverse frequency balancing were 76.63%, 83.68%, and 86%, and using uniform class balancing was 77%, 84%, and 86% respectively. While, the dice score for these regions was calculated as 78%, 85%, and 87%, respectively using the median frequency balancing. These results clearly demonstrate that the median frequency balancing could better adjust the class weights in favor of classes with smaller volumes such as enhanced tumors. Without using any balancing technique, the DSC for the above regions was 76%, 84%, and 88% correspondingly, which depicts a bending in favor of larger classes volume.
      Deeplabv3+ needs a feature extraction (encoding) network and uses its weights for the initialization of the model. To assess the role of Resnet18, which was employed as the main feature extractor in encoder part of Deeplabv3+ in this work, a model without Resnet18 weights was executed on the training dataset. This procedure was performed based on the scratch method using app deep network designer in MATLAB2021a software and removing the Resnet18 layers and their weights from the layer graph of the network and replacing them with the same layers with no weights. Results of this examination showed a DSC of 10%, 13%, and 18% for ET, TC and WT respectively, which are very weak in comparison with the optimized model results, i.e. our Deep-Net. Other possible networks, such as Resnet50 and Xception which can be used as an extractor in the combination of Deeplabv3+, were also examined using the reported hyper-parameters. The results of the comparison of three pre-trained networks (Resnet18, Resnet50, and Xception) were illustrated in Table 3. As results indicate the combination of Deeplabv3+ and Resnet18 using the obtained hyper-parameters demonstrated superior performance for glioblastoma tumor segmentation. The required time and memory are much lower in the training process of employing the Deeplabv3+ with ResNet-18 backbone.
      Table 3Comprehensive comparison of three pre-trained networks: Resnet18, Resnet50 and Xception, to segment the different tumor regions.
      Networks

      Deeplabv3+

      [Resnet50]

      Downsample = 8
      Deeplabv3+

      [Resnet18]

      Downsample = 8
      Deeplabv3+

      [Xception]

      Downsample = 16
      Deeplabv3+

      [Resnet18]

      Downsample = 16
      Scores
      DSC for ET region68.4%78%50.3%77.4%
      DSC for TC region76.6%85%60.4%84.5%
      DSC for WT region79.5%87%66.8%85.5%
      Validation Accuracy96%97.5%93.2%97%
      Validation Loss0.180.140.330.15
      Run Time [min]1741274683196
      * ET: Enhanced Tumor, TC: Tumor Core, WT: Whole Tumor.

      Discussions

      In this study, a new deep learning-based algorithm, named Deep-Net was developed and optimized for segmentation of the glioblastoma tumor in brain MR images. The purpose of the study was to segment four main classes of the whole brain. The diagonal elements of the confusion matrix, indicate the high sensitivity for all considered classes. The results indicate that the developed and trained Deep-Net can reliably and confidently segment all tumor sub-regions. The good achieved results in this work could be due to two reasons: 1- appropriate training dataset, and 2-intensive hyper-parameters optimization process. The data used for training in this study is around 35,000 MR images from two different sequences. This large dataset helps the network to adequately learn the pattern of the data set and recognize the border of each region. In addition, training the network with a large dataset and also optimizing the epoch number assure us that the trained network is not over-fitted, which is avouched by all CCF and loss function curves. The used dataset in this study was limited to two MRI sequences (T1ce and FLAIR), each MRI sequence can help to extract special information about the tumor and its sub-regions. Two informative MR images are contrast-enhanced T1-weighted, which provide good information about the enhanced tumor region; and also, the FLAIR MR images, which provide the appropriate information about the edema area. Combination of these two modalities and neglecting the other MRI sequences such as T2 and T1 from the BraTS dataset helped us to reduce the memory and time required in the training process without losing the important information [
      • Zhao X.
      • Wu Y.
      • Song G.
      • Li Z.
      • Fan Y.
      • Zhang Y.
      Brain tumor segmentation using a fully convolutional neural network with conditional random fields.
      ], and also to avoid the network confusion between inverse image intensities of different MRI sequences. This issue can be confirmed by comparing the results of this work with the results of other investigations that used more MRI sequences in their training process, see Table 4.
      Table 4Comparison of the results of this study with previous investigations
      MethodSummary of methodDatabaseMRI ModalityGradeDice Score
      Whole TumorTumor CoreEnhancing Tumor
      Ref#
      • Zikic D.
      • Ioannou Y.
      • Brown M.
      • Criminisi A.
      Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
      Convolutional Neural NetworksBraTS 2013 challengeT1, T2, T1c, FlairHGG847469
      Ref#
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      A CNN with small 3 × 3 kernelsBRATS 2013 LeaderboardT1, T2, T1c, FlairHGG887673
      BRATS 2013 TrainingT1, T2, T1c, FlairHGG888377
      Ref#
      • Vaidhya K.
      • Thirunavukkarasu S.
      • Alex V.
      • Krishnamurthi G.
      Multi-modal brain tumor segmentation using stacked denoising autoencoders.
      stacked denoising autoencoder (SDAE),)A deep neural network that reconstructs its input(BRATS 2015 ChallengeT1, T2, T1c, FlairHGG816864
      Ref#
      • Dong H.
      • Yang G.
      • Liu F.
      • Mo Y.
      • Guo Y.
      Automatic brain tumor detection and segmentation using u-net based fully convolutional networks.
      U-Net Based Fully Convolutional NetworksBRATS 2015 TrainingT1, T2, T1c, FlairHGG888781
      Ref#
      • Yi D.
      • Zhou M.
      • Chen Z.
      • Gevaert O.
      3-D convolutional neural networks for glioblastoma segmentation.
      3-D ConvolutionBRATS 2015 TrainingT1, T2, T1c, FlairHGG897680
      Ref#
      • Cui S.
      • Mao L.
      • Jiang J.
      • Liu C.
      • Xiong S.
      Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
      Deep Cascaded Neural NetworkBRATS 2015 TrainingT1, T2, T1c, FlairHGG908181
      Ref#
      • Randhawa R.S.
      • Modi A.
      • Jain P.
      • Warier P.
      Improving boundary classification for brain tumor segmentation and longitudinal disease progression.
      Deep neural networkBRATS 2016 ChallengeT1, T2, T1c, FlairHGG787679
      Ref#
      • Shaikh M.
      • Anand G.
      • Acharya G.
      • Amrutkar A.
      • Alex V.
      • Krishnamurthi G.
      Brain tumor segmentation using dense fully convolutional neural network.
      Dense Fully Convolutional Neural NetworkBraTS2017

      Challenge
      T1, T2, T1c, FlairHGG848380
      Ref#
      • Kim G.
      Brain tumor segmentation using deep fully convolutional neural networks.
      Deep Fully ConvolutionalBRATS 2017

      Training
      T1, T2, T1c, FlairHGG878280
      Ref#
      • Ahmad P.
      • Qamar S.
      • Hashemi S.R.
      • Shen L.
      Hybrid Labels for Brain Tumor Segmentation.
      3D U-Net with

      residual-dense ASPP
      BraTS2019

      Training
      T1, T2, T1c, FlairHGG878478
      Deep-NetDeeplabv3 + With Resnet-18BRATS 2020 TrainingT1c, FlairHGG878578
      The training process of the Deep-Net was repeated many times to reach the optimum values for a few hyper-parameters, which significantly affect the network segmentation performance. The optimization of each hyper-parameter was widely investigated in this study, which is rarely performed in previous studies. The most effecting hyper-parameters that were assessed, and optimized in this study were BS, ILR, LRDF, drop period, L2R rate, down sample rate and the epoch. As mentioned before the 37th epoch is the epoch chosen as the best epoch. This choice ensures that the optimum point is sufficiently far from the increasing part of the curve, without imposing a big increase in the runtime. Continuing the training process after that epoch, imposes more cost than benefit to the network, as the accuracy improvement has reached the plateau. The training process was implemented with the following optimized hyper-parameters: BS = 64, ILR = 0.009, LRDF = 0.4, drop period = 10, L2 regularization rate = 0.005, down sample rate = 8, and epoch number of 37. The achieved CCF with the optimum training process was 97.5 with a loss of 0.143.
      The used Deeplabv3+ structure benefits from ASPP convolution to generate the denser feature maps and improve the preservation of the spatial information for the feature maps. This property leads to a better localization and consequently affects the segmentation process. The acceptable results reported in previous studies and current work confirm the performance of the ASPP in segmentation tasks [

      Moreno Lopez M, Ventura J. Dilated convolutions for brain tumor segmentation in MRI scans. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:253–62. 10.1007/978-3-319-75238-9_22.

      ,

      Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

      ,
      • Ahmad P.
      • Qamar S.
      • Hashemi S.R.
      • Shen L.
      Hybrid Labels for Brain Tumor Segmentation.
      ,
      • Choudhury A.R.
      • Vanguri R.
      • Jambawalikar S.R.
      • Kumar P.
      Segmentation of brain tumors using DeepLabv3+.
      ,
      • Chen L.-C.
      • Zhu Y.
      • Papandreou G.
      • Schroff F.
      • Adam H.
      Encoder-decoder with atrous separable convolution for semantic image segmentation.
      ]. As aforementioned, Deep-Net is a deep neural network based on the Deeplabv3+ structure which employs Resnet18 as its main feature extractor (in the encoder part), with substantial modifications and optimization. Resnet18 is a powerful and instructive network to extract contextual features from medical images [

      Alex V, Khened M, Ayyachamy S, Krishnamurthi G. Medical image retrieval using Resnet-18 for clinical diagnosis 2019;1095410:35. 10.1117/12.2515588.

      ]. Residual networks use the idea of “Skip Connection” and can skip some of the network’s layers to simplify the network and uses fewer layers in the early stages of training [
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ]. This technique speeds up the learning process by reducing vanishing gradients through fewer propagation layers. Thus, this pre-trained network may represent a good trade-off between depth, that takes computational time, and precision, and was selected as the main CNN in encoder part of the Deep-Net. Recruiting the Resnet18 as a feature extractor had a significant effect on improving the results of semantic segmentation of Deep-Net. Results of removing the weights of Resnet18 from the developed structure and using other pre-trained networks instead of Resnet18, which was reported in results section, highlight the role of initial weights of pre-trained Resnet18 in Deep-Net performance and speed. The results of previous investigations that used other possible combinations of Deeplabv3+ and different backbones approve this conclusion. Choudhury et al. [
      • Brosch T.
      • Tang L.Y.W.
      • Yoo Y.
      • Li D.K.B.
      • Traboulsee A.
      • Tam R.
      Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation.
      ] tried to combine different MRI modalities to discover the best combination of MR images, thus they generated 18 combinations of four modalities of MR images and trained a model based on the Deeplabv3 + and Xception network as its backbone. The best obtained DSC for their research was 71%, 79%, and 88% for the ET, TC, and WT regions respectively. As their DSC results indicate their constructed Deeplabv3+ with Xception couldn’t recognize the enhanced tumor and tumor core as well as Deeplabv3+ with the incorporation of Resnet18 used in our work. Chen et al. [
      • Menze B.H.
      • Jakab A.
      • Bauer S.
      • Kalpathy-cramer J.
      • Farahani K.
      • Kirby J.
      • et al.
      ] also used the Deeplabv3+ architecture along the Xception and Resnet101. Their reported performance of 89% for Xception (in comparison with the Deep-Net performance=97.5%) confirms the above-mentioned conclusion about the incorporation of Deeplabv3+ and Resnet18. They reported that the performance of the Deeplabv3+ and Xception combination was 2% superior to utilizing Resnet101 as the backbone network [
      • Menze B.H.
      • Jakab A.
      • Bauer S.
      • Kalpathy-cramer J.
      • Farahani K.
      • Kirby J.
      • et al.
      ].
      In this study, the average time needed to segment a brain slice by the constructed Deep-Net (on GPU, NVIDIA GeForce RTX 3080 with 12 GB RAM, in addition to CPU Intel Core i7-10700 K (@3.5 GHz) with 32 GB RAM on the Windows 10 system) is 0.03 sec, thus generation of the segmentation maps takes less than 5 sec for whole-brain examination with 155 slices. It seems that this time is acceptable and fast enough for an automatic segmentation algorithm, especially in contrast to the previously reported time for one slice segmentation. However, the reported time for different algorithms has been calculated on various hardware and computing platforms, so a decisive judgment between different models’ speeds is a bit difficult. For example, Cui et al. [
      • Cui S.
      • Mao L.
      • Jiang J.
      • Liu C.
      • Xiong S.
      Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
      ] reported a time of 0.98 sec for one slice segmentation using an FCN-8 s structure and 1.54 sec recruiting a two-stage algorithm. Both algorithms were executed on dual Intel E5-2603 CPUs and a middle-end GPU graphic card (GeForce GTX 1080, NVIDIA). Dong et al. [
      • Dong H.
      • Yang G.
      • Liu F.
      • Mo Y.
      • Guo Y.
      Automatic brain tumor detection and segmentation using u-net based fully convolutional networks.
      ] developed a deep algorithm that required 2–3 sec time for a slice segmentation. Their model was a U-Net-based fully convolutional network and performed on an NVIDIA Titan X (Pascal) GPU with 12G memory. Other researchers published computational times for their models such as 30 sec for a Multi-Scale 3D CNN with fully connected CRF on an NVIDIA GTX Titan X GPU, 3G RAM [
      • Kamnitsas K.
      • Ledig C.
      • Newcombe V.F.J.
      • Simpson J.P.
      • Kane A.D.
      • Menon D.K.
      • et al.
      Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.
      ], 25 sec for TwoPathCNN and 3 min for InputCascadeCNN on an NVIDIA TITAN black GPU [
      • Havaei M.
      • Dutil F.
      • Pal C.
      • Larochelle H.
      • Jodoin P.
      A Convolutional Neural Network Approach to Brain Tumor Segmentation.
      ] and 8 min for a deep CNN on a GPU NVIDIA GeForce GTX 980 equipped on an Intel Core i7 3.5 GHz [
      • Pereira S.
      • Pinto A.
      • Alves V.
      • Silva C.A.
      Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
      ], to predict a complete tumor study. Thus, it can be concluded that our Deep-Net is also time-efficient, besides its acceptable performance. However, a few previously constructed networks [
      • Cui S.
      • Mao L.
      • Jiang J.
      • Liu C.
      • Xiong S.
      Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
      ,
      • Dong H.
      • Yang G.
      • Liu F.
      • Mo Y.
      • Guo Y.
      Automatic brain tumor detection and segmentation using u-net based fully convolutional networks.
      ,
      • Yi D.
      • Zhou M.
      • Chen Z.
      • Gevaert O.
      3-D convolutional neural networks for glioblastoma segmentation.
      ] reported slightly better segmentation performance than Deep-Net, but they are slower than the proposed algorithm in this work. The DSC result of this algorithm for enhancing tumor, tumor core and the whole tumor was compared with other works in order to show how this algorithm may improve HGG brain tumor segmentation performance. Table 4 summarized the DSC score of some recently reported constructed deep networks. All listed works in the table are trained by the MR images of the HGG patients. The Deep-Net was trained using only two more informative MRI sequences while others in the table were trained with three or four different MRI sequences. For example, Ahmad et al. [
      • Ahmad P.
      • Qamar S.
      • Hashemi S.R.
      • Shen L.
      Hybrid Labels for Brain Tumor Segmentation.
      ] used a structure based on the ASPP and 3D U-Net and their reported DSC results are approximately the same as the obtained results for Deep-Net, despite using a 2D CNN structure and only two MRI sequences data in our model instead of 3D CNN structure and four MRI sequences data. Similarly, Yi et al. [
      • Yi D.
      • Zhou M.
      • Chen Z.
      • Gevaert O.
      3-D convolutional neural networks for glioblastoma segmentation.
      ] employed data of four MRI sequences and a 3D CNN for segmentation of HGG brain tumor, as listed in the Table 4, their reported DSC for WT and ET classes is 2% more than the calculated score for Deep-Net, however, the Deep-Net outperforms it for TC class. In the literature, the association of T1 and FALIR images of the BraTS2020 dataset has been similarly employed [

      Ahuja S, Panigrahi BK, Gandhi TK. Fully automatic brain tumor segmentation using DeepLabv3+ with variable loss functions. Proc 8th Int Conf Signal Process Integr Networks, SPIN 2021 2021:522–6. 10.1109/SPIN52536.2021.9566128.

      ] on a Deeplabv3+ with different baseline networks such as MobileNetv2. This research reports a global accuracy of 96%. which is lower than Deep-Net global accuracy (97.5%). Deep cascade CNN showed relatively better performance for WT class, whereas Deep-Net performed better for ET and TC classes. Cui et al [
      • Cui S.
      • Mao L.
      • Jiang J.
      • Liu C.
      • Xiong S.
      Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
      ] used a BraTS2015 dataset and a two-step structure. In the first step, the tumor region was localized in each MRI slice, and then in the next step a CNN with deeper architecture and smaller kernels was recruited to classify HGG brain tumor into multiple sub-regions. They obtained a DSC of 71%, 76% and 88% for ET, TC and WT coordinately. Marcinkiewicz et al. [
      • Ronneberger O.
      • Fischer P.
      • U-net T.B.
      Convolutional networks for biomedical image segmentation.
      ] recruited a cascade multimodal U-net on BraTS2018 dataset (combined HGG and LGG) and achieved a DSC of 75%, 81% and 89% for the above classes. Kermi et al. [
      • Hu X.
      • Li H.
      • Zhao Y.
      • Dong C.
      • Menze B.H.
      • Piraud M.
      Hierarchical multi-class segmentation of glioma images using networks with multi-level activation function.
      ] used the BraTS2018 dataset (combined HGG and LGG) employing a CNN architecture similar to U-net with essential modification on it. They reached a DSC of 72%, 79% and 87% for these tumor sub-regions. As can be understood from the Table 4 and the previously reported scores, the DSC of the Deep-Net in segmentation of ET, TC, and WT classes (78%, 85%, and 87%) show a promising result to ensure that the Deep-Net structure can be accurate and reliable enough to be used in glioblastoma segmentation studies [

      Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

      ].
      Despite the acceptable performance of the Deep-Net, some defects are observed in the segmented maps in some slices of test data. These defects may occur for classes with small areas, which leads to a large amount of type I error for such regions, and the Deep-Net is not successful in the classification of the pixels in the correct class. These defects are more notable when the enhanced tumor area is very small and thin. Another possible error that may occur in the segmented maps is the generalization and integration of the pixels in the periphery of that class. This defect is mostly observed in peri-tumoral edema and tumor necrotic core.

      Conclusion

      The results and experiments presented in this paper describe an end-to-end system for multi-modal segmentation of glioblastoma tumor sub-regions, normal tissue, and background based on a deep learning algorithm. The main contributions made in this work are finding the optimal and effective hyper-parameters to construct an adaptive and intelligence segmentation network. The ablation studies, which were performed in this study, clearly demonstrated the roll of the optimal values for hyper-parameters of a deep network on its performance. A combination of Deeplabv3 + as main structure and Resnet18 as its initial weights worked well in GBM tumor segmentation from T1ce and FLAIR MRI modalities images. Recently, our team showed [

      Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

      ] that the trained Deep-Net can perform well in GBM tumor segmentation from DCE-MR images too. Thus, it can be claimed that the high sensitivity (accuracy) and high precision (reliably reproducible) for all classes reveal that the trained Deep-Net in this work is accurate enough and maybe used as a reliable help for neuro-radiologists in clinical segmentation practice. Manual delineation of the tumors is a user-dependent and very time-consuming process that should be automated through the development of an automatic and computerized segmentation system. Deep learning-based segmentation of MR images offers the potential for accurate definition of the complex treatment volumes and speeding the contouring process in radiotherapy treatment planning. Automatic segmentation can be used to present a reproducible, and standard segmentation protocol that helps the physicians of different facilities throughout the world in the effective treatment of tumors.

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      References

        • Pirtoli L.
        • Gravina G.L.
        • Giordano A.
        Radiobiology of Glioblastoma: Recent Advances and Related Pathobiology.
        Humana Press Totowa, NJ2016: 305
        • Zikic D.
        • Ioannou Y.
        • Brown M.
        • Criminisi A.
        Segmentation of Brain Tumor Tissues with Convolutional Neural Networks.
        Proc MICCAI-BRATS. 2014; : 36-39
        • Vaidhya K.
        • Thirunavukkarasu S.
        • Alex V.
        • Krishnamurthi G.
        Multi-modal brain tumor segmentation using stacked denoising autoencoders.
        BrainLes 2015, Springer. 2015; : 181-194
        • Havaei M.
        • Dutil F.
        • Pal C.
        • Larochelle H.
        • Jodoin P.-M.
        A convolutional neural network approach to brain tumor segmentation.
        BrainLes 2015, Springer. 2015; : 195-208
        • Randhawa R.S.
        • Modi A.
        • Jain P.
        • Warier P.
        Improving boundary classification for brain tumor segmentation and longitudinal disease progression.
        Int Work Brainlesion Glioma, Mult Sclerosis, Stroke Trauma Brain Inj, Springer. 2016; : 65-74
        • Shaikh M.
        • Anand G.
        • Acharya G.
        • Amrutkar A.
        • Alex V.
        • Krishnamurthi G.
        Brain tumor segmentation using dense fully convolutional neural network.
        Springer, Int. MICCAI brainlesion Work.2017: 309-319
        • Kim G.
        Brain tumor segmentation using deep fully convolutional neural networks.
        Springer, Int. MICCAI Brainlesion Work.2017: 344-357
      1. Moreno Lopez M, Ventura J. Dilated convolutions for brain tumor segmentation in MRI scans. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:253–62. 10.1007/978-3-319-75238-9_22.

        • Papadimitroulas P.
        • Brocki L.
        • Christopher Chung N.
        • Marchadour W.
        • Vermet F.
        • Gaubert L.
        • et al.
        Artificial intelligence: Deep learning in oncological radiomics and challenges of interpretability and data harmonization.
        Phys Medica. 2021; 83: 108-121https://doi.org/10.1016/j.ejmp.2021.03.009
        • Castiglioni I.
        • Rundo L.
        • Codari M.
        • Di Leo G.
        • Salvatore C.
        • Interlenghi M.
        • et al.
        AI applications to medical images: From machine learning to deep learning.
        Phys Medica. 2021; 83: 9-24https://doi.org/10.1016/j.ejmp.2021.02.006
        • Nadeem M.W.
        • Al Ghamdi M.A.
        • Hussain M.
        • Khan M.A.
        • Khan K.M.
        • Almotiri S.H.
        • et al.
        Brain tumor analysis empowered with deep learning: A review, taxonomy, and future challenges.
        Brain Sci. 2020; 10: 1-33https://doi.org/10.3390/brainsci10020118
        • Dehkordi A.N.V.
        • Kamali-Asl A.
        • Ewing J.R.
        • Wen N.
        • Chetty I.J.
        • Bagher-Ebadian H.
        An adaptive model for rapid and direct estimation of extravascular extracellular space in dynamic contrast enhanced MRI studies.
        NMR Biomed. 2017; 30: e3682
      2. Shukla G, Alexander GS, Bakas S, Nikam R, Talekar K, Joshua D. Advanced magnetic resonance imaging in glioblastoma : A review Advanced magnetic resonance imaging in glioblastoma : a review. Chin Clin Oncol 2017:40. 10.21037/cco.2017.06.28.

        • Zegers C.M.L.
        • Posch J.
        • Traverso A.
        • Eekers D.
        • Postma A.A.
        • Backes W.
        • et al.
        Current applications of deep-learning in neuro-oncological MRI.
        Phys Medica. 2021; 83: 161-173https://doi.org/10.1016/j.ejmp.2021.03.003
        • Dehkordi A.N.V.
        • Kamali-Asl A.
        • Wen N.
        • Mikkelsen T.
        • Chetty I.J.
        • Bagher-Ebadian H.
        DCE-MRI prediction of survival time for patients with glioblastoma multiforme: using an adaptive neuro-fuzzy-based model and nested model selection technique.
        NMR Biomed. 2017; 30: e3739
        • Rüfenacht E.
        • Jungo A.
        • Ermiş E.
        • Blatti-Moreno M.
        • Hemmatazad H.
        • Aebersold D.M.
        • et al.
        Organs at Risk Delineation for Brain Tumor Radiation Planning in Patients with Glioblastoma Using Deep Learning.
        Int J Radiat Oncol Biol Phys. 2019; 105: E718-E719
      3. Inmaculada M, Ocaña G, Román KL, Urzelai NL, Ángel M, Ballester G, et al. Deep Learning in Healthcare. vol. 171. 2020.

      4. Dehkordi A, Sina S, Khodadadi F. A Comparison of Deep Learning and Pharmacokinetic Model Selection Methods in Segmentation of High-Grade Glioma. Front Biomed Technol 2021;8. 10.18502/fbt.v8i1.5858.

        • Mazzara G.P.
        • Velthuizen R.P.
        • Pearlman J.L.
        • Greenberg H.M.
        • Wagner H.
        Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation.
        Int J Radiat Oncol Biol Phys. 2004; 59: 300-312
        • Pereira S.
        • Pinto A.
        • Alves V.
        • Silva C.A.
        Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images.
        IEEE Trans Med Imaging. 2016; 35: 1240-1251https://doi.org/10.1109/TMI.2016.2538465
        • Wang G.
        • Li W.
        • Ourselin S.
        • Vercauteren T.
        Automatic Brain Tumor Segmentation Based on Cascaded Convolutional Neural Networks With Uncertainty Estimation.
        Front Comput Neurosci. 2019; 13: 1-13https://doi.org/10.3389/fncom.2019.00056
        • Ahmad P.
        • Qamar S.
        • Hashemi S.R.
        • Shen L.
        Hybrid Labels for Brain Tumor Segmentation.
        Springer, Int. MICCAI Brainlesion Work.2019: 158-166
        • Kamnitsas K.
        • Ledig C.
        • Newcombe V.F.J.
        • Simpson J.P.
        • Kane A.D.
        • Menon D.K.
        • et al.
        Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.
        Med Image Anal. 2017; 36: 61-78https://doi.org/10.1016/j.media.2016.10.004
        • Isensee F.
        • Kickingereder P.
        • Wick W.
        • Bendszus M.
        • Maier-Hein K.H.
        No new-net.
        Springer, Int. MICCAI Brainlesion Work.2018: 234-244
        • Myronenko A.
        3D MRI brain tumor segmentation using autoencoder regularization.
        Springer, Int. MICCAI Brainlesion Work.2018: 311-320
        • Havaei M.
        • Davy A.
        • Warde-Farley D.
        • Biard A.
        • Courville A.
        • Bengio Y.
        • et al.
        Brain tumor segmentation with Deep Neural Networks.
        Med Image Anal. 2017; 35: 18-31https://doi.org/10.1016/j.media.2016.05.004
        • Cui S.
        • Mao L.
        • Jiang J.
        • Liu C.
        • Xiong S.
        Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network.
        J Healthc Eng. 2018; 2018
        • Havaei M.
        • Dutil F.
        • Pal C.
        • Larochelle H.
        • Jodoin P.
        A Convolutional Neural Network Approach to Brain Tumor Segmentation.
        In BrainLes. 2016; 1: 195-208https://doi.org/10.1007/978-3-319-30858-6
        • McKinley R.
        • Wepfer R.
        • Gundersen T.
        • Wagner F.
        • Chan A.
        • Wiest R.
        • et al.
        Nabla-net: A deep dag-like convolutional architecture for biomedical image segmentation.
        Int Work Brainlesion Glioma, Mult Sclerosis, Stroke Trauma Brain Inj, Springer. 2016; : 119-128
        • Li Y.
        • Shen L.
        Deep learning based multimodal brain tumor diagnosis.
        Springer, Int. MICCAI Brainlesion Work.2017: 149-158
        • Hu X.
        • Li H.
        • Zhao Y.
        • Dong C.
        • Menze B.H.
        • Piraud M.
        Hierarchical multi-class segmentation of glioma images using networks with multi-level activation function.
        Springer, Int. MICCAI Brainlesion Work.2018: 116-127
        • Brosch T.
        • Tang L.Y.W.
        • Yoo Y.
        • Li D.K.B.
        • Traboulsee A.
        • Tam R.
        Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation.
        IEEE Trans Med Imaging. 2016; 35: 1229-1239
        • Ronneberger O.
        • Fischer P.
        • U-net T.B.
        Convolutional networks for biomedical image segmentation.
        Int Conf Med Image Comput Comput Interv. 2015; 234–241https://doi.org/10.1109/ACCESS.2021.3053408
        • Menze B.H.
        • Jakab A.
        • Bauer S.
        • Kalpathy-cramer J.
        • Farahani K.
        • Kirby J.
        • et al.
        The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). 2014; 0062: 1-33https://doi.org/10.1109/TMI.2014.2377694
      5. Pawar K, Chen Z, Shah NJ, Egan G. Residual encoder and convolutional decoder neural network for glioma segmentation. Lect Notes Comput Sci (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 2018;10670 LNCS:263–73. 10.1007/978-3-319-75238-9_23.

        • Kermi A.
        • Mahmoudi I.
        • Khadir M.T.
        Deep convolutional neural networks using U-Net for automatic brain tumor segmentation in multimodal MRI volumes.
        Springer, Int. MICCAI Brainlesion Work.2018: 37-48
        • Choudhury A.R.
        • Vanguri R.
        • Jambawalikar S.R.
        • Kumar P.
        Segmentation of brain tumors using DeepLabv3+.
        Springer, Int. MICCAI Brainlesion Work.2018: 154-167
        • Marcinkiewicz M.
        • Nalepa J.
        • Lorenzo P.R.
        • Dudzik W.
        • Mrukwa G.
        Segmenting brain tumors from MRI using cascaded multi-modal U-Nets.
        Springer, Int. MICCAI Brainlesion Work.2018: 13-24
        • Chen L.-C.
        • Zhu Y.
        • Papandreou G.
        • Schroff F.
        • Adam H.
        Encoder-decoder with atrous separable convolution for semantic image segmentation.
        Proc Eur Conf Comput Vis. 2018; : 801-818
        • Bakas S.
        • Akbari H.
        • Sotiras A.
        • Bilello M.
        • Rozycki M.
        Data Descriptor : Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features.
        Nat Publ Gr. 2017; 4: 1-13https://doi.org/10.1038/sdata.2017.117
        • Bakas S.
        • Reyes M.
        • Jakab A.
        • Bauer S.
        • Rempfler M.
        • Crimi A.
        • et al.
        Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge.
        ArXiv Prepr ArXiv181102629. 2018;
        • Li C.
        • Gore J.C.
        • Davatzikos C.
        Multiplicative intrinsic component optimization (MICO) for MRI bias fi eld estimation and tissue segmentation.
        Magn Reson Imaging. 2014; 32: 913-923https://doi.org/10.1016/j.mri.2014.03.010
        • Gallivanone F.
        • D’Ambrosio D.
        • Carne I.
        • D’Arcangelo M.
        • Montagna P.
        • Giroletti E.
        • et al.
        A tri-modal tissue-equivalent anthropomorphic phantom for PET, CT and multi-parametric MRI radiomics.
        Phys Med. 2022; 98: 28-39https://doi.org/10.1016/j.ejmp.2022.04.007
      6. MATLAB 2021a, The MathWorks Inc. Deep Learning Toolbox™. Natick, Massachusetts, United States; 2021.

        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Spatial Pyramid Pooling in Deep Convolutional.
        IEEE Trans Pattern Anal Mach Intell. 2014; 37: 346-361
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition.
        Proc IEEE Conf Comput Vis pattern Recognit. 2016; : 770-778
        • Sandler M.
        • Howard A.
        • Zhu M.
        • Zhmoginov A.
        • Chen L.C.
        MobileNetV2: Inverted Residuals and Linear Bottlenecks.
        Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2018; : 4510-4520https://doi.org/10.1109/CVPR.2018.00474
        • Xception C.F.
        Deep learning with depthwise separable convolutions.
        Proc IEEE Conf Comput Vis pattern Recognit. 2017; : 1251-1258
      7. Tan M, Le Q V. EfficientNet: Rethinking model scaling for convolutional neural networks. 36th Int Conf Mach Learn ICML 2019 2019;2019-June:10691–700.

        • Bischke B.
        • Helber P.
        • Borth D.
        • Dengel A.
        Segmentation of imbalanced classes in satellite imagery using adaptive uncertainty weighted class loss.
        IGARSS 2018–2018 IEEE Int Geosci Remote Sens Symp, IEEE. 2018; : 6191-6194
        • Tian J.
        • Mithun N.
        • Seymour Z.
        • Chiu H.-P.
        • Kira Z.
        Striking the Right Balance.
        Recall Loss for Semantic Segmentation. 2021;
      8. Ayhan B, Kwan C. Tree, shrub, and grass classification using only RGB images. Remote Sens 2020;12. 10.3390/RS12081333.

        • Zhang Z.
        Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels.
        32nd Conf Neural Inf Process Syst. 2018
        • Zhao M.
        • Xin J.
        • Wang Z.
        • Wang X.
        • Wang Z.
        Interpretable Model Based on Pyramid Scene Parsing Features for Brain Tumor MRI Image Segmentation.
        Comput Math Methods Med. 2022; 2022https://doi.org/10.1155/2022/8000781
        • Shirish Keskar N.
        • Mudigere D.
        • Nocedal J.
        • Smelyanskiy M.
        • Tang P.T.P.
        On large-batch training for deep learning: Generalization gap and sharp minima.
        ArXiv E-Prints. 2016:arXiv-1609.;
        • Shamir R.R.
        • Duchin Y.
        • Kim J.
        • Sapiro G.
        • Harel N.
        C ontinuous Dice Coefficient : a M ethod for Evaluating P robabilistic S egmentation s.
        ArXiv Prepr ArXiv190611031. 2018; : 1-4
        • Csurka G.
        • Larlus D.
        • Perronnin F.
        • Meylan F.
        What is a good evaluation measure for semantic segmentation?.
        BMVC. 2013; 27: 10-5244
        • Zhao X.
        • Wu Y.
        • Song G.
        • Li Z.
        • Fan Y.
        • Zhang Y.
        Brain tumor segmentation using a fully convolutional neural network with conditional random fields.
        Int Work Brainlesion Glioma, Mult Sclerosis, Stroke Trauma Brain Inj, Springer. 2016; : 75-87
      9. Alex V, Khened M, Ayyachamy S, Krishnamurthi G. Medical image retrieval using Resnet-18 for clinical diagnosis 2019;1095410:35. 10.1117/12.2515588.

        • Dong H.
        • Yang G.
        • Liu F.
        • Mo Y.
        • Guo Y.
        Automatic brain tumor detection and segmentation using u-net based fully convolutional networks.
        Springer, Annu. Conf. Med. image Underst. Anal.2017: 506-517
        • Yi D.
        • Zhou M.
        • Chen Z.
        • Gevaert O.
        3-D convolutional neural networks for glioblastoma segmentation.
        ArXiv Prepr ArXiv161104534. 2016;
      10. Ahuja S, Panigrahi BK, Gandhi TK. Fully automatic brain tumor segmentation using DeepLabv3+ with variable loss functions. Proc 8th Int Conf Signal Process Integr Networks, SPIN 2021 2021:522–6. 10.1109/SPIN52536.2021.9566128.