If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Istituto di Chimica dei Composti OrganoMetallici, Consiglio Nazionale delle Ricerche, Florence, ItalyEuropean Laboratory For Non Linear Spectroscopy, Università degli Studi di Firenze, Florence, Italy
Dipartimento di Fisica e Astronomia, Università degli Studi di Firenze, Florence, ItalyScuola di Scienze della Salute Umana, Università degli Studi di Firenze, Florence, Italy
Dipartimento di Fisica e Astronomia, Università degli Studi di Firenze, Florence, ItalyScuola di Scienze della Salute Umana, Università degli Studi di Firenze, Florence, Italy
Pattern Analysis & Computer Vision, Istituto Italiano di Tecnologia, Genoa, ItalyDipartimento di Ingegneria Navale, Elettrica, Elettronica e delle Telecomunicazioni, Università degli Studi di Genova, Genoa, Italy
Pattern Analysis & Computer Vision, Istituto Italiano di Tecnologia, Genoa, ItalyDipartimento di Ingegneria Navale, Elettrica, Elettronica e delle Telecomunicazioni, Università degli Studi di Genova, Genoa, Italy
Pattern Analysis & Computer Vision, Istituto Italiano di Tecnologia, Genoa, ItalyDipartimento di Ingegneria Navale, Elettrica, Elettronica e delle Telecomunicazioni, Università degli Studi di Genova, Genoa, Italy
Dipartimento di Fisica e Astronomia, Università degli Studi di Firenze, Florence, ItalyScuola di Scienze della Salute Umana, Università degli Studi di Firenze, Florence, Italy
Dipartimento di Fisica e Astronomia, Università degli Studi di Firenze, Florence, ItalyIstituto Nazionale di Fisica Nucleare - Sezione di Firenze, Sesto Fiorentino, Florence, Italy
An “Ad hoc” phantom is designed and used to collect a large database of CT images.
•
Convolutional Neural Netwoks (CNNs) for denoise and segmentation tasks are developed.
•
Quality evaluation of CNNs processing is achieved by conventional and radiomic approaches.
•
Performances of different multi-tasks CNNs are investigated.
•
Radiomic features are proposed as indicators of texture alterations induced by CNNs.
Purpose
We investigate, by an extensive quality evaluation approach, performances and potential side effects introduced in Computed Tomography (CT) images by Deep Learning (DL) processing.
Method
We selected two relevant processing steps, denoise and segmentation, implemented by two Convolutional Neural Networks (CNNs) models based on autoencoder architecture (encoder-decoder and UNet) and trained for the two tasks. In order to limit the number of uncontrolled variables, we designed a phantom containing cylindrical inserts of different sizes, filled with iodinated contrast media. A large CT image dataset was collected at different acquisition settings and two reconstruction algorithms. We characterized the CNNs behavior using metrics from the signal detection theory, radiological and conventional image quality parameters, and finally unconventional radiomic features analysis.
Results
The UNet, due to the deeper architecture complexity, outperformed the shallower encoder-decoder in terms of conventional quality parameters and preserved spatial resolution. We also studied how the CNNs modify the noise texture by using radiomic analysis, identifying sensitive and insensitive features to the denoise processing.
Conclusions
The proposed evaluation approach proved effective to accurately analyze and quantify the differences in CNNs behavior, in particular with regard to the alterations introduced in the processed images. Our results suggest that even a deeper and more complex network, which achieves good performances, is not necessarily a better network because it can modify texture features in an unwanted way.
International Commission On Radiological Protection (ICRP). ICRP PUBLICATION 26: 1977 Recommendations of the International Commission on Radiological Protection. Ann ICRP 1977;26(1(3)).
], generating an increasing demand for visual examination by the radiologist. This need, together with the evolution of Artificial Intelligence (AI) technologies supported by the increasing availability of computational resources, has driven the scientific research towards the development of AI-based tools. In particular, various Deep Learning (DL) models have been designed by researchers for relevant tasks applied to CT images and have been largely documented in literature because of their potential support to the radiologist staff in the diagnosis from clinical images [
Causey J, Guan Y, Dong W, Walker K, Qualls J, Prior F, et al. Lung cancer screening with low-dose CT scans using a deep learning approach. arXiv:190600240 2019;.
Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. Artificial intelligence and machine learning in radiology: Opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol 2018;15(3, Part B):504–508. doi: 10.1016/j.jacr.2017.12.026.
]. Indeed, Convolutional Neural Networks (CNNs) have shown remarkable effectiveness in several tasks, such as automatic localization and segmentation of low contrast objects and denoising of clinical images [
Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. Artificial intelligence and machine learning in radiology: Opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol 2018;15(3, Part B):504–508. doi: 10.1016/j.jacr.2017.12.026.
Causey J, Guan Y, Dong W, Walker K, Qualls J, Prior F, et al. Lung cancer screening with low-dose CT scans using a deep learning approach. arXiv:190600240 2019;.
Humphries T, Si D, Coulter S, Simms M, Xing R. Comparison of deep learning approaches to low dose CT using low intensity and sparse view data. Proc SPIE 2019;10948. doi: https://doi.org/10.1117/12.2512597.
Kim H, Park CM, Lee M, Park SJ, Song YS, Lee J, et al. Impact of Reconstruction Algorithms on CT Radiomic Features of Pulmonary Tumors: Analysis of Intra- and Inter-Reader Variability and Inter-Reconstruction Algorithm Variability. PLoS One 2016;11. doi: https://doi.org/10.1371/journal.pone.0164924.
Many studies have evaluated the impact of DL algorithms on image quality by means of conventional image analysis, including estimation of signal to noise ratio (SNR), noise power spectrum (NPS) and modulation transfer function (MTF) ([
Greffier J, Hamard A, Pereira F, Barrau C, Pasquier H, et al. Image quality and dose reduction opportunity of deep learning image reconstruction algorithm for CT: a phantom study. Eur Radiol 2020;30:3951–3959. doi: https://doi.org/10.1007/s00330-020-06724-w.
Low-Dose Abdominal CT Using a Deep Learning-Based Denoising Algorithm: A Comparison with CT Reconstructed with Filtered Back Projection or Iterative Reconstruction Algorithm.
Samei E, Richard S. Assessment of the dose reduction potential of a model-based iterative reconstruction algorithm using a task-based performance metrology. Med Phys 2015;42. doi: https://doi.org/10.1118/1.4903899.
Deep learning-based reconstruction in ultra-high-resolution computed tomography: Can image noise caused by high definition detector and the miniaturization of matrix element size be improved?.
Greffier J, Hamard A, Pereira F, Barrau C, Pasquier H, et al. Image quality and dose reduction opportunity of deep learning image reconstruction algorithm for CT: a phantom study. Eur Radiol 2020;30:3951–3959. doi: https://doi.org/10.1007/s00330-020-06724-w.
]). However, beyond the surprisingly good results in many domains, DL tools are black-boxes, which interpretation remains still obscure. As a consequence the images resulting from DL processing might present subtle alterations that might impact further processing steps. As a matter of facts, the nature and the extent of possible alterations of the information content in output images after the CNNs processing remains mostly unexplored.
The aim of this work is, therefore, to address the problem of possible alterations induced by DL processing methods on CT images. We performed such investigation by means of simplified images obtained from a specially designed homogeneous PolyMethyl MethAcrylate (PMMA) phantom with inserts of various diameters and contrasts. Indeed, one strength of this research consists in having collected a large amount of labeled CT images necessary to train and test the CNNs in highly controlled conditions.
We focused the analysis on two tasks: denoising and object segmentation. Indeed these two tasks are exemplar of the two main issues in the field of image evaluation: detectability preservation with increasing noise and areas identification in low contrast environment. All experiments were based on the adoption of two CNN architectures with different complexity: a standard encoder-decoder and its extension, the more powerful UNet model, largely applied in the field of medical imaging for segmentation and denoising tasks [
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing. p. 234–241, ISBN 978-3-319-24574-4; 2015.
Yu S, Park B., Jeong J. Deep iterative down-up cnn for image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2019.
Yuan H, Jia J, Zhu Z. Sipid: A deep learning framework for sinogram interpolation and image denoising in low-dose ct reconstruction. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 2018, p. 1521–1524. doi: https://doi.org/10.1109/ISBI.2018.8363862.
Bao L, Yang Z, Wang S, Bai D, Lee J. Real image denoising based on multi-scale residual dense block and cascaded u-net with block-connection. In: Proc. of the IEEE/CVF Conf. CVPR. 2020.
]. The choice of using these two models was guided by the intention to study how the increasing computational power affects the structure of information in the processed images. In addition, since there are some works reporting increased performance of CNNs addressing multi-tasks learning [
Quattoni A, Collins M, Darrell T. Transfer learning for image classification with sparse prototype representations. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, p. 1–8. doi: 10.1109/CVPR.2008.4587637.
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
], we explored this solution modifying the models is such a way to solve both denoising and segmentation in one shot.
CNNs outputs were assessed through various metrics. For the segmentation results the Dice Similarity Coefficient (DSC) and the area under Receiving Operating Characteristic (ROC) curves [
Hajian-Tilaki K. Receiver operating characteristic (roc) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med 2013;4(2):627–635. doi: PMID:24009950; PMCID: PMC3755824.
] were adopted. Denoising results were assessed by means of conventional metrics (SNR, NPS, MTF). In addition, since radiomics is attracting interest because of its descriptive power [
Midya A, Chakraborty J, Gönen M, Do RKG, Simpson AL. Influence of CT acquisition and reconstruction parameters on radiomic feature reproducibility. J of Med Imaging 2018;011020. doi: https://doi.org/10.1117/1.JMI.5.1.011020.
]. In this analysis we exploited the radiomics features with a novel perspective, considering them as “radiomics properties” characterizing the information structure. The comparison of images in terms of these “radiomics properties” allowed us to identify the “sensitive features”, i.e. the features affected by the CNN denoising process, hence, more suitable for analyzing the alterations introduced by such process. This analysis allowed to better characterize the image alterations, adding further information to the one arising from conventional metrics. On the other hand, the same comparison allowed us to identify the “robust features”, i.e. those insensitive to the CNN denoise process.
The investigation procedure we adopted was found efficient enough to accurately detect and quantify the differences in CNNs behavior both for the attained results and for the alterations introduced in the processed images. Its sensitivity has also proved adequate to properly quantify the differences in noise shape associated to the reconstruction methods: Filtered Back Projections (FBP) and Iterative Reconstruction (IR) [
Influence of sinogram affirmed iterative reconstruction of ct data on image noise characteristics and low-contrast detectability: An objective approach.
We designed and manufactured a PMMA phantom (Fig. 1) of ellipsoidal shape consisting of four adjacent blocks (size cm each); one block is homogeneous in PMMA to produce background images; two blocks contain 5 cylindrical inserts each, with increasing diameters (3 mm, 4 mm, 5 mm, 6 mm, 7 mm) and 5 cm high; the inserts were filled with iodinated contrast media at two different concentrations to obtain signal differences of 80 HU ( contrast) and 70 HU ( contrast) with respect to background (125 HU) at 120 kV. In the fourth block four solid cylindrical inserts of different materials (acrylic, Teflon, polyzene, polyvinylchloride) and variable diameters were placed, in order to measure low and high contrast spatial resolution.
Fig. 1Lateral view (a) and top view (b) of one of the two blocks containing 5 inserts filled with iodinated contrast media; (c) Picture of the fourth block, containing 4 solid inserts for MTF evaluation.
Acquisition for CNNs training and optimization was carried out by using a 128 slice CT scanner (Somatom Definition Flash, Siemens Healthcare) and selecting the standard oncological protocol for abdomen
(Helical, 120 kVp, 200 mAs ref, acquisition mm, collimation mm, pitch, scan Field of View (FoV) cm, rotation time 0.5 s, CareDose 4D = on, Care kV = off)
. In Fig. 2 two pictures of the acquisition setup are shown.
Fig. 2Pictures of the acquisition setup in the CT room.
CT scans of the entire phantom were performed at 8 + 1 different current settings (Table 1), in order to vary the image noise and to obtain clean (HD) reference images to generate ground truth for the neural network training. Table 1 reports the volumetric CT dose index (here named CTDI) for the current acquisition settings.
Table 1Current settings for CT acquisitions, and corresponding expected image quality level (HD stands for High Dose index, the highest quality).
Reconstruction was performed by applying two different algorithms: FBP (convolution kernel B41s) and IR (SAFIRE, strength 3, convolution kernel IF41s). For each block of the phantom, 24 slices each 2 mm thick were taken into account. In order to have only one contrast object in each image, reconstructions were performed by setting Reconstructed FoV (RFoV) equal to . Fig. 3 shows an image of the entire phantom and two images at different radiation levels.
Fig. 3a) Image (IR method) of the entire phantom acquisition (RFoV=); b) and c) reconstructions (IR method) around an insert (RFoV ) of the phantom acquired at high dose index (CTDI ) and low dose index (CTDI ), respectively. The square in (a) identified the insert represented in (b) and (c).
We describe here our CNN approach for CT images processing. The relative background on machine learning (ML) and deep learning (DL) can be found in S.I. material (Section S2). All the computational codes were developed using Python programming language.
2.2.1 Data preprocessing
In order to set up the large number of images required both to train and validate the CNNs, we reconstructed multiple independent ( RFoV) images from the CT acquisitions of the phantom by positioning the RFoV, in HD images, in correspondence of each contrast insert and then repeating the reconstruction by randomly varying the center coordinates of the RFoVs (10 times for each insert) in order to generate more images containing the contrast objects in different locations (see also Fig. S1 in S.I. - Supplementary Information - material). Reconstruction of images from the acquisitions at the other 8 CTDI values was carried out by setting the same coordinates of the RFoV centers. Having selected a slice thickness of 2 mm, we were able to reconstruct 24 images, for each insert along the z axis of the phantom with the same coordinates as mentioned above, therefore totaling images containing a contrast object. Same amount of images was obtained without any contrast object, from the homogeneous block of the phantom. At the end of this step each scan contains superimpoised slices where the insert is always in the same position. In order to reduce overfitting (see Deep learning background Section S2 in S.I. and Ref. [
Cogswell M, Ahmed F, Girshick BR, Zitnick LC, Batra D. Reducing overfitting in deep networks by decorrelating representations. International conference on learning representations; 2015.
] to each slice, i.e. to the subset of 24 images reconstructed over the depth of the phantom (therefore with the same coordinates) consisting of 90 degrees rotation and flipping. These augmentation operations add further variation on the inserts position from one slice to the other, in order to reduce the number of images containing the object in the same position. By repeating the procedure for both FBP and IR we obtained two separate dataset. We fed the neural network with the described dataset divided between train and validation data in ratio.
Ground truths for the denoise task were generated from the highest CTDI scan by averaging all the slices over the entire depth of the inserts (24 slices), in order to obtain very clean images with minimum noise. Binary masks for denoise metrics computation were then obtained from these ground truth images by means of a Gaussian adaptive thresholding algorithm (function adaptiveThreshold() of the Imgproc class of Open Computer Vision – OpenCV library [
]). Ground truths for segmentation task were obtained from such binary masks: firstly the coordinates of the center of the contrast object were determined and thereafter a binary image was generated locating in those coordinates the center of a disk with a diameter equal to the corresponding nominal one in the phantom.
2.2.2 CNNs architecture
We developed a neural network architecture to perform multiple tasks, namely segmentation and denoise of the reconstructed phantom CT images (input images). In order to get insight into the behavior of the trained neural network in manipulating the input images, we implemented and tested different models, based on autoencoder architecture (Fig. 4), and then we evaluated their performances on the considered tasks. The achievement of the two tasks simultaneously was obtained by inserting two parallel branches (one for segmentation and one for denoise) at the end of the decoder step of the model. In this way we realized a double-task model, that recently have attracted interest in literature for the good performances reached in both the tasks trained [
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
]. Detailed schemes of CNN models used can be found in S.I. (Section S2).
Fig. 4A schematic version of our architecture inspired by UNet: a noisy image (left) is given as input and the model tries to reconstruct the clean image (bottom-right) and to highlight the position of the object (top-right).
An encoder-decoder (Enc-Dec) consisting in 4 convolutional layers for encoding, followed by 2 fully connected layers interposed with dropout layers (dropout rate tuned to reach optimum at 0.1), after which the model splits into two branches made of three convolutional layers each, for optimization of the two tasks.
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing. p. 234–241, ISBN 978-3-319-24574-4; 2015.
] adapted to the current tasks: a combination of max pooling, convolutional and fully connected layers for a total of 12 layers and 3 skip connections. Skip connections concatenate high resolution features produced by encoder to upsampled features of decoder to enable precise segmentation. The two branches for the separated tasks consist of three convolutional and two additional concatenation layer each.
•
The UNet model was trained also separately in the two tasks of segmentation and denoise, by minimizing only the corresponding loss at one time. We will address these trained model as UNet-den and UNet-seg.
Kanakis M, Bruggemann D, Saha S, Georgoulis S, Obukhov A, Gool LV. Reparameterizing convolutions for incremental multi-task learning without task interference. In: Proc. ECCV 2020; vol. 12365. Springer; 2020, p. 689–707. doi: 10.1007/978-3-030-58565-5_41.
Ponomarenko N, Krivenko S, Egiazarian K, Astola J, Lukin V. Weighted mse based metrics for characterization of visual quality of image denoising methods. 2009.
Liu D, Wen B, Liu X, Huang TS. When image denoising meets high-level vision tasks: A deep learning approach. CoRR 2017;abs/1706.04284. URL http://arxiv.org/abs/1706.04284.
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
], a linear combination of two losses was implemented for the training: a mean square error loss and a binary cross entropy loss to optimize denoise and segmentation tasks, respectively. The relative weight of the two losses was tuned using validation set. Further details on loss tuning process and optimizer algorithm (Adam) can be found in S.I. (Section S2).
Training was carried out from scratch. Model’s hyperparameters were fine-tuned using validation set. We employed Tensorflow framework [
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. TensorFlow: Large-scale machine learning on heterogeneous systems. 2015. doi: http://tensorflow.org/.
] and GPU parallel programming to build and train the CNN models. All experiments were run on a Nvidia GeForce RTX 2080 Ti GPU with CUDA 10 support.
2.2.3 Performance metrics
We used the DSC to evaluate the spatial performances of the trained models in localizing the inserts and their shape. ROC curves were computed to address the behavior of the algorithms in radiological terms. Finally, the area under the ROC curves (AUC) was computed as a function of CTDI for the two tasks to estimate the overall neural network performance by means of such a common metrics used in diagnostic imaging. Additional description of the metrics implemented can be found in S.I. (Section S2.).
2.3 Conventional images analysis
Conventional image quality parameters quantifying detectability (via SNR), noise frequency content (via NPS), spatial resolution (through MTF) were evaluated on original and denoised images and then compared. The ground truth images were used to automatically define the insert region as the region of interest (ROI) by means of pixel aggregation and adaptive threshold which maximize the SNR. Details on the above mentioned quality indices and their extraction procedures are reported in Section S3 of the S.I.
2.4 Radiomic features extraction
A Python based code was developed to extract features from every image using the open-source package Pyradiomics [
]. All features are computed by setting a fixed bin width equal to 25, after applying intensity normalization of the images in the pre-processing step.
For the study of the denoise process we concentrated on second order statistics features, which provide information on the texture of the examined region (see Table S1 in S.I. section for the complete list of these features). Both homogeneous images and those with insert were considered and a square ROI of fixed size was defined for each image to select a background region, not including insert; the size was equal to pixels, the largest satisfying these conditions on all considered images.
At first, we identified the features most relevant and indicative for the characterization of noise and denoising process: these specific features were selected in terms of repeatability and sensitivity [
]. Repeatability refers to features that remain the same for the entire dataset of images acquired in the same CT acquisition conditions while the term sensitivity refers to the variability of a feature in relation to different image acquisition settings, or different reconstruction techniques.
Repeatability was quantified by means of the coefficient of variation (CV),i.e., the ratio of the standard deviation to the absolute value of the mean, evaluated on each subset of original images at various CTDI values (from 4.4 to 20 mGy): the smaller the CV the better the repeatability. We arbitrarily assumed as repeable those features with CV % for all CTDI values, a threshold value halfway between 10% and 20% that are often chosen to categorize features into groups according to their coefficient of variation [
]. In order to investigate the sensitivity to noise of radiomic features, we plotted their average values, evaluated on the subsets of original images, as a function of CTDI; a linear fit was applied to the data (normalized by max) and the extrapolated slope parameter was taken as an estimate of the sensitivity to noise patterns.
We have also examined the sensitivity of each feature to different reconstruction methods, defined as the mean percentage difference between the values extracted from the original images reconstructed by the FBP and IR techniques: results show that, in general, those features sensitive to noise are also the most sensitive to the reconstruction method.
At last, all features were grouped in terms of their Spearman rank correlation [
] have been considered redundant, and therefore interchangeable.
The features representative of each redundant group, i.e. those with the highest sensitivity to noise, have been selected as candidates of sensitive characterization: ShortRunEmphasis and LongRunEmphasis are features that respectively measure the distribution of the lengths of short and long runs, i.e. those 1-dimensional (1D) structures of consecutive pixels with the same gray level value; decrease of ShortRunEmphasis and/or increase of LongRunEmphasis are indices of coarser 1D noise texture. The ZonePercentage is a feature which quantifies 2-dimensional (2D) pattern and low values correspond to coarser 2D texture. Busyness belongs measures variations between adjacent pixels; as the Busyness decreases the spatial frequency of intensity changes decreases. For the mathematical definition of the above features see [
For each feature the percentage difference between the values extracted by denoised and corresponding original images was computed; feature varying on average by less than were considered “robust” [
Oliver J, Budzevich M, Hunt D, Moros EG, Latifi K., TJ T.J.D., et al. Sensitivity of Image Features to Noise in Conventional and Respiratory-Gated PET/CT Images of Lung Cancer: Uncorrelated Noise Effects. Technol Cancer Res Treat 2017;16(5) 595–608. doi: https://doi.org/10.1177/1533034616661852.
Two unrelated shape features, Sphericity (that corresponds to roundness in 2D) and MeshSurface, have been extracted for the spatial and geometrical characterization of the denoising and segmentation processing: the former is the ratio of the perimeter of the object under analysis and the perimeter of a circle with the same surface area of the object; the latter estimates the surface area of the object [
An example of the neural network output for the Enc-Dec and UNet models is shown in Fig. 5, in case of an original image, reconstructed via IR technique, containing an insert of 5 mm diameter with contrast. Different background textures are produced by the two models, and it is noticeable the better spatial accuracy of the UNet model respect to Enc-Dec in segmenting the insert, probably due to the presence of skip connections within the model architecture. Analogous examples for FBP reconstruction, in case of input images containing insert or background-only, are presented in Fig. S6 and S7 of S.I., respectively.
Fig. 5Example of outputs from Enc-Dec and UNet trained models in denoising and segmenting the same original image (from validation dataset), reconstructed via IR technique and containing an insert of 5 mm diameter filled with contrast.
Figs. 6 a) and b) present plots of the computed DSCs as a function of CTDI, for inserts of different sizes and contrasts, in case of both denoise and segmentation tasks performed by the UNet model.
Fig. 6DSCs as a function of CTDI, computed (a) on binary masks generated from UNet model denoised images and (b) from UNet model segmented images. Solid and dotted lines refer to inserts of and contrast respectively. Insert diameters range from 3 to 7 mm. Average error bars (equal to the standard deviation of the average) for each curve are shown in the bottom, indicating larger errors for small inserts diameters.
We combined signal detection theory (i.e. DSC metric) with ROC analysis, this latter being the preferred method to assess image quality in diagnostic radiology. AUC, computed from ROC curves, increases with CTDI (Fig. 7), with saturation at around 8 mGy and values above 0.95.
Fig. 7Area under ROC curves for UNet segmentation and denoise, computed separately for FBP and IR reconstructions.
In order to better understand the CNNs behavior in addressing the two tasks and their mutual influence on the processed images, we carried out two separate trainings by minimizing a single loss, thus optimizing one single task at one time. Comparison between the UNet double task and single tasks (UNet-den and UNet-seg), and quantification of the superior performances of the UNet respect to the Enc-Dec, were carried out by evaluating AUC as a function of CTDI (Fig. 8).
Fig. 8AUC as a function of CTDI, computed on the images output of the trained models: UNet (blu curves) and Enc-Dec (green curves) are the double-task models, and in the parenthesis it is indicated the output images subset used to compute AUC (denoised -solid line- or segmented -dashed line- images); UNet-den and UNet-seg are the single task models, which produce, respectively, denoised (red solid line) and segmented (red dashed line) images used to compute AUC. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
SNR was computed for all original images, reconstructed via FBP and IR techniques, and for the denoised images produced by the trained CNN models. The results are summarized in Fig. 9 where SNR mean values are reported as a function of CTDI.
Fig. 9SNR as a function of CTDI, calculated on original images (FBP reconstruction) containing the inserts with contrasts C1 and on denoised images produced by the different models. Lines are linear fits to emphasize the data trends. Note that a break is introduced on y axis, to better show the original’s values. The errors bars, barely visible, are the Standard Deviation of the represented mean values. SNR in case of IR method is presented in Fig. S11 of S.I.
NPS was evaluated for original and ground truth images and for the corresponding denoised images produced by the trained CNNs, within a background region ( pixels) where no insert is present. The results are shown in Fig. 10 for original images at CTDI = 4.4 mGy (no signicative difference has been found as a function of noise level).
Fig. 10NPS curves of the ground truth -GT- (green), original (blue) and denoised images produced by the trained CNNs: Enc-Dec (orange), UNet (black) and UNet-den (pink). A break on y axis is used to show the UNet and UNet-den remarkable peak near zero. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
MTFs were evaluated from the inserts in the ground truth images and from the images containing the solid inserts (no remarkable differences were found, as shown in Fig. S12 of S.I. section); we verified that the average operation to obtain ground truth from single slices didn’t degrade spatial resolution (Fig. S.13 of S.I. section). These reference MTFs were then compared to those obtained from the same inserts in the denoised images produced by the UNet and UNet-den models. The MTF curves from the 7 mm diameter insert and C1 contrast are shown in Fig. 11; very similar results were obtained for all the other inserts of different size and contrast.
Fig. 11MTF curves computed from the ground truth (GT) images (black) and the denoised output images of the CNN UNet (red) and UNet-den (pink). The error bars represent the Standard Deviation of the mean values. The test images used have insert with 7 mm diameter and C1 contrast. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
The circular-edge technique for the extraction of MTFs is not generally applicable to the Enc-Dec denoised images since therein inserts roundness is not well preserved, as discussed in the next section (see Fig. 15a).
3.3 Radiomic features analysis
The repeatable and sensitive radiomic features selected by mean of the procedure discussed in Section 2.4 are listed in Table 2, along with the corresponding repeatability and sensitivity indices. The distributions of the four selected features are shown in Fig. 12, in case of original images (FBP reconstruction at the lower dose index), ground truths and corresponding denoised images.
Table 2List of features sensitive to denoise process. The maximum correlation coefficient between these features is 0.58. Coefficient of variation measured from the sample of ground truth images reconstructed by FBP method. Slope parameter of the line fitted to the average values of the single feature as a function of CTDI for FBP method. Mean percentage difference between FBP and IR reconstruction methods for images acquired with quality level 8.
Fig. 12Distributions of features listed in Table 2, corresponding to dose index mGy and reconstruction method FBP. The line spanning the full width of the box corresponds to the median of the distribution while the square box extends from 25 to 75 percentile values and the vertical bars represent the range of distribution not including outliers (matplotlib.pyplot.boxplot function)
In Fig. 13 the values of ShortRunEmphasis feature, extracted from the denoised images and from the original images, are plotted as a function of the CTDI.
Fig. 13ShortRunEmphasis feature mean values versus the CTDI, normalized to the maximum, from original and denoised images. Lines are fits of the corresponding data. The error bars represent the Standard Deviation of the mean values.
In Fig. 14 is reported the mean percentage difference between denoised and original images for all the radiomic features with respect to the denoising process. The values of all the standard features [
] were calculated from each subset (8 different dose indices) of input images and compared to those extracted from the corresponding CNNs processed images. In this way the features less sensitive to noise and to the denoise process (i.e. more robust) were identified (Fig. 14). Different robust features were identified depending on the CNN models, based on the criterion described in 2.4 section. A summary table with the overall results is reported in S.I. (Table S2).
Fig. 14Mean percentage difference between denoised and original images for all radiomic features and for three neural networks. The values are sorted in decreasing order on UNet. The four features displayed correspond to those listed in Table 2 and the robust features are listed in Table S2 in S.I.
Fig. 15Distributions of Sphericity (a) and MeshSurface (b) features extracted from segmented images, corresponding to CTDI = 4.4 mGy and 7 mm diameter insert with C1 contrast, FBP reconstruction: comparison among different models output. The horizontal dashed line in panel represents the MeshSurface value computed on the ground truth images for segmentation task. See fig. 12 for the graphical representation of the values.
The distributions of the shape features, Sphericity and MeshSurface, for the different trained models are summarized in Fig. 15, while the detailed variation of the Sphericity respect to dose index, inserts size and contrasts for the UNet segmentation is shown in Fig. 16.
Fig. 16Sphericity as a function of noise from trained UNet model, computed on segmented images. Solid lines indicate inserts of C2 contrast of different diameters while dotted lines indicates inserts of C1 contrast. Average error bars (equal to the standard deviation of the average) for each curve are shown in the bottom, indicating larger errors for small inserts diameters.
The UNet model trained on Siemens dataset was finally tested in the denoise and segmentation tasks of images from a different dataset, acquired on the same phantom with a different CT scanner from Philips, to estimate the capacity of generalization of the trained model and to open new discussion about the complex interplay of factors involved in the CNNs optimization process and therefore in the quantitative evaluation of CNNs behavior. The description of the acquisition method and data preprocessing, as well as a full overview of the metrics evaluation computed on the UNet output images is reported in S.I. (Section S7).
The comparison between the performances of the trained CNN on validation (Siemens) and test (Philips) dataset, in terms of AUC, are shown in Fig. 17 as a function of CTDI.
Fig. 17AUC as a function of CTDI computed to compare the performances of the UNet model in the processing of images from two different datasets. One (acquired on CT scanner of Siemens vendor - green lines) used for validation after training, and a second one (acquired in a CT scanner from Philips vendor - red lines) used only to test the domain generalization. The plots indicate the performance in the denoise (solid line) and segmentation (dashed line) tasks. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
The results of the deep learning experiments were very promising, in terms of both the capacity of trained CNN models to correctly locate the contrast objects within the background pattern, and the quality of the denoised output images. As it can be inferred from Fig. 5, UNet model learned to discriminate the presence of the inserts even in condition of high noise level, where by eye it is difficult to identify the edge of the object, or even to recognize the presence of the object itself.
Different alteration of the background region are observed between the denoised imaged produced by Enc-Dec and UNet model: the Enc-Dec seems to partially preserve the noise texture of the ground truth images, while the UNet model strongly flatten the homogeneous region around the insert. These alterations have been attributed to the mean square error loss function [
Ponomarenko N, Krivenko S, Egiazarian K, Astola J, Lukin V. Weighted mse based metrics for characterization of visual quality of image denoising methods. 2009.
Liu D, Wen B, Liu X, Huang TS. When image denoising meets high-level vision tasks: A deep learning approach. CoRR 2017;abs/1706.04284. URL http://arxiv.org/abs/1706.04284.
]. However, we found substantial difference between the two models, and we believe that beyond spatial performances, also this kind of alterations must be taken under consideration when dealing with CNNs image processing, and should influence the choice of the model. Therefore we can state that even a complex network, which achieves good performances, is not necessary a better network because it can modify texture features in an unwanted way. A quantitative explanation of this result is found in the NPS evaluation and in the radiomic features analysis reported below.
AUC as a function of CTDI (Fig. 7) shows, as expected, increased performances at larger CTDI, with saturation trend observed above 8 mGy (for the UNet model). It is worth noticing that slightly better performance was achieved with the IR method rather than with FBP reconstruction. This last results is of particular interest considering that the quantification of CT image quality has become a non-trivial issue after the introduction of IR techniques, which alter the images in non-linear way and have shown noise dependent spatial resolution ([
Baker M, Dong F, Primak A, Obuchowski N, Einstein D, Gandhi N, et al. Contrast-to-Noise Ratio and Low-Contrast Object Resolution on Full- and Low-Dose MDCT: Safire Versus Filtered Back Projection in a Low-Contrast Object Phantom and in the Liver. AJR Am J Roentgenol 2012;199. doi: https://doi.org/DOI:10.2214/AJR.11.7421.
Observer Performance in the Detection and Classification of Malignant Hepatic Nodules and Masses with CT Image-Space Denoising and Iterative Reconstruction.
Comparison between different models allows to understand the influence of the architectures on their behavior and to address the contribution of the model layers in the training performance. Even if the autoencoder scheme is the starting point for both the Enc-Dec and the UNet model (see schemes in Fig. S2 of S.I. section), in the first case the low variability of the layers types (only convolutional and fully connected) and the small number of layers were the main reasons for the lower performance observed. Furthermore, the Enc-Dec suffered from overfitting, and its generalization ability was not satisfactory (Enc-Dec losses trend in Fig. S4 in S.I.).
On the contrary, in case of the UNet model, the greater depth (i.e. the large number of concatenated layers) and the presence of multiple skip connections, which prevent the losses of fundamental information during the encoding step, give rise to enhanced performances (see metrics comparison in Fig. 8) and reduce the overfitting issue (UNet losses trend in Fig. S3 in S.I.). The trained UNet model not only locates the insert correctly in most cases, but also associates the correct diameter and shape, as deductible from Fig. 6, where DSC scores are plotted as a function of CTDI and confirmed by the Sphericity feature analysis in Fig. 16. DSC curves also indicate, as expected, that the trained neural network performs better in case of low noise and for images containing inserts with larger diameters and more pronounced contrast.
The experiments performed by training one task at a time (either denoise or segmentation), by minimizing the two losses separately, allow to evaluate the reciprocal influence of the two branches during training. The comparison of AUC (Fig. 8) shows that denoise performances of the UNet model, when optimized for both tasks, are slightly superior respect to the model trained for the denoise task (UNet-den) alone. On the contrary, segmentation performances improve with the single task training: this means that in our experiments segmentation task is useful to improve denoise but denoise doesn’t help segmentation, differently than in differeprevious studies ([
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
]). The reciprocal reinforcement of the two tasks is confirmed by the evaluation, shown in Fig. S10 of S.I., of the performances of the UNet-den and UNet-seg in the task for which they were not trained, i.e. denoise for UNet-seg and segmentation for UNet-den.
The tuning of the relative weights seems to confirm the previous result. The evaluated AUC as a function of (cross entropy loss for segmentation) weights (Fig. S4 of S.I.) suggests that the two tasks are correlated: the denoise performances indeed are strongly quenched not only at large weight, which is expected considering that (mean square error loss for denoise) becomes negligible, but also at very low weights, indicating that the optimization of model weights to improve segmentation is useful also to achieve better denoise performance.
Interesting results were found when testing the deep learning algorithm on a dataset of images acquired by a different CT scanner. The estimated metrics, reported in Section S7 of S.I., evidence DSC score mostly above 50%, except the smaller inserts (3 mm diameter), where the model lacks of spatial identification ability. AUC values are also above 50%. This is a surprisingly good result when considering the lack of specific training of the CNN. It is noticeable that CNN performances increase with CTDI until 10 mGy (the largest CTDI of the training dataset), then they start to decrease even if the larger SNR should facilitate the inserts detection, possibly because no training at all was performed in that range of CTDIs. This behavior confirms a well-known generalization issue associated to CNNs: their optimization involve a combination of multiple factors. Disentanglement of such factors is a complex subject that is often overcome by increasing the dataset variability in the training step.
In order to perform a proper comparison between the performances of the trained CNN on such different datasets, several factors should be taken into account. First of all, SNR, noise intensity and texture are influenced not only by CTDI, but also by the reconstruction algorithm, proper of each CT scanner, and by the concentrations of the iodinated contrast media. The complexity of the generalization aspects requires a dedicated, extended, investigation which is outside the goal of the present paper.
4.2 Quality evaluation of the CNNs performance by means of conventional metrics
The results shown in Fig. 9 demonstrate the very high performance of the CNNs in reducing the images noise. Actually in the processed images the SNR increases by factor 20–30 respect to the original ones, well above , the reference value of the Rose detectability criterion [
] and therefore providing a quantitative explanation of the sharp visual detectability of the low SNR insert shown in Fig. 5. While the SNRs computed on original images increase with CTDI and contrast, as expected, in almost all the denoised images, the SNRs are essentially independent from CTDI and contrast. This can be considered a further valuable performance of the CNNs. Both UNet and UNet-den perform better than the Enc-Dec, the latter showing however a little lower dependence from the contrast of the inserts. The SNR of the single task UNet-den slightly overtakes the one of the combined denoise-segmentation UNet.
The NPS curves (Fig. 10) show the expected shape when computed on original images. A very high zero-frequency peak is clearly visible in the NPS obtained from the UNet and UNet-den output images, while the ones obtained from Enc-Dec output images (as well as from ground truth images) show a smaller zero peak according to its weaker capability of reducing noise (see also Fig. 5 and Fig. 12). This result is in agreement with the already stated behavior of the considered CNNs.
UNet denoised images provide an MTF curve almost identical to the ground truth images unlike common noise reduction methods that can result in a loss of spatial resolution (e.g. a typical smoother kernel) [
Artmann U, Wueller D. Interaction of image noise, spatial resolution, and low contrast fine detail preservation in digital image processing. In: Digital Photography V; vol. 7250. SPIE; 2009, p. 154–162. doi: 10.1117/12.805927.
]. This ability to preserve spatial resolution has already been observed in case of UNet-based models and it has been attributed to the presence of global skip connections [
], we avoided using max pooling to preserve details, implementing instead stride in convolutions to reduce feature maps size in the contracting path. In the expanding path, features of contracting path are concatenated. This improves reconstruction, mostly the concatenation of first layers features to the last ones, that has been shown to preserve in a better way spatial resolution.
In the single task UNet-den processed images the spatial resolution is slightly negatively affected, confirming that denoise can benefit from the presence of the simultaneous segmentation task. For the Enc-Dec the calculation of spatial resolution is obfuscated by the noticeable geometrical distortion of the edge of the inserts as previously discussed.
4.3 Quality evaluation of the CNNs performance by means of radiomic features
The 1D and 2D texture features, listed in Table 2 and reported in Fig. 12, show that for all the trained models the noise reduction process does not produce a texture similar to that of the ground truth images (especially for the UNet models). In addition, such alteration is dependent on the CNN model.
The values of the four selected features computed on the denoised images suggest a finer noise texture and a greater spatial frequency of intensity changes, respect to the ground truth images.
Furthermore, the distributions of the features values obtained from the denoised images are much wider than those obtained from the ground truth and original images, suggesting that CNNs produce variability in the properties of the noise texture.
The denoising process of Enc-Dec is very different from that of the UNet, since alteration of the texture is less pronounced and the values of all the estimated features are more similar to the ground truth images respect to UNet (see Fig. 14). On the other hand, UNet greatly reduces noise by producing a much more homogeneous texture: this result reflect the NPS analysis and it is apparent also by looking at Fig. 5. This characteristic should be taken under consideration when dealing with the applications of CNNs processing to clinical images: such remarkable alteration of the background pattern could produce unusual images for the experienced eye of the clinical staff which may compromise their interpretation and therefore may not be easily accepted.
UNet and UNet-den behave very similarly to each other, with a slightly more pronounced alteration of the texture for the first model, as shown in Fig. 12 for ShortRunEmphasis and ZonePercentage features.
The dependence of the ShortRunEmphasis feature on CTDI (Fig. 13) is less significant for CNNs denoised images respect to original images: this suggests that the background noise pattern in the denoised images remains quite the same regardless of the dose index of input images, in analogy with the trend found for the SNR (Fig. 9).
The features that are not altered by the CNNs images processing, identified by the robustness analysis reported in Fig. 14, should be taken under consideration for a potential clinical application of the AI algorithms: those features are indeed the best suitable for radiomic texture investigation of CT images under different acquisition conditions.
The distributions of the geometrical features (Fig. 15) suggest that both denoise and segmentation processes alter the shape of the contrast object. The Enc-Dec neural network alters the shape of the inserts more than the UNet does, by decreasing their area and by degrading their roundness, which makes it difficult to evaluate the MTF curve, as already discussed in Section 3.3. No significant performance differences were observed between the UNet and the UNet-den, that presents slightly lower performances at high noise level.
The Sphericity feature, computed on UNet segmented images and plotted in Fig. 16 as a function of CTDI for different inserts, indicates that for small objects diameters less than 4 mm the performances significantly worsen.
5. Conclusions
We developed, trained and tested two CNNs, namely a standard encoder-decoder and its extension, the Unet, in the tasks of segmentation and denoise of simple CT images obtained by scanning a specifically designed PMMA phantom, with the aim of determine in a quantitative way the different behavior of the two models and, at the same time, the different textures they induced into the processed images.
The alterations produced by the CNN algorithms on the original images was thoroughly studied by combining deep learning metrics, such as Dice Similarity Coefficient and area under Receiving Operating Characteristic curves, conventional metrics (SNR, NPS, MTF) and radiomics features.
As expected UNet attains superior results in the segmentation task however in the denoise task, in spite of an apparent much higher definition of the test object, it introduced also evident alterations of the background, whereas the Enc-Dec was able to reproduce the ground truth more faithfully.
It is worth noticing that when ideally extrapolating this proposition towards the clinical applications of images denoised via CNNs, an alert occurs, that is to pay attention to the selection of the appropriate CNN not to introduce heavy alterations into the noise texture that could mislead the proper judgment of the radiologist.
The highly controlled conditions of our experiments allowed us to give some reliable contribution to the sometimes debated problem of the possible reciprocal influence of the model branches, each designed for one individual tasks, when combined them together for multi-tasks learning. We reported a slight increase in denoise performance, contrary to what previously reported [
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
], suggesting that there is still space for additional experiments in this field. In any case, either in the single or in the multi-task learning, images denoised by means of the UNet model preserved the original spatial resolution, despite the trade-off between denoise and spatial resolution commonly reported in literature [
Artmann U, Wueller D. Interaction of image noise, spatial resolution, and low contrast fine detail preservation in digital image processing. In: Digital Photography V; vol. 7250. SPIE; 2009, p. 154–162. doi: 10.1117/12.805927.
The approach adopted was effective, albeit in simplified images, to accurately detect and quantify the differences in the behavior of CNNs both for the result obtained from the tasks and for the alterations introduced in the processed images.
On the other hand by using the radiomic approach we were able to select a number of robust features, i.e. features insensitive to the CNN images processing with both models. For this reason they constitute potential candidates for a conventional radiomic analysis of images processed by diverse CNNs.
The extremely good results of the UNet model in the segmentation task of the CT images of the phantom persuaded us about the actual feasibility of a CNN based model observer [
Gong H, Hu Q, Walther A, Koo C, Takahashi E, Levin D, et al. Deep-learning-based model observer for a lung nodule detection task in computed tomography. Proc SPIE 2020;042807. doi: https://doi.org/10.1117/1.JMI.7.4.042807.
Reith F, Wandell B. Comparing pattern sensitivity of a convolutional neural network with an ideal observer and support vector machine. arXiv:191105055 2019;.
Alnowami M, Mills G, Awis M, Elangovanr P, Patel M, Halling-Brown M, et al. A deep learning model observer for use in alterative forced choice virtual clinical trials. Proc SPIE 2018;10577. doi: https://doi.org/10.1117/12.2293209.
] behaving like a human observer for the highly needed task of optimizing (detectability vs patient dose) current CT protocols.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The authors acknowledge the support and the CT resources provided by the Radiology department of the Careggi University Hospital in Florence, Italy.
Supplementary data
The following are the Supplementary data to this article:
International Commission On Radiological Protection (ICRP). ICRP PUBLICATION 26: 1977 Recommendations of the International Commission on Radiological Protection. Ann ICRP 1977;26(1(3)).
Causey J, Guan Y, Dong W, Walker K, Qualls J, Prior F, et al. Lung cancer screening with low-dose CT scans using a deep learning approach. arXiv:190600240 2019;.
Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. Artificial intelligence and machine learning in radiology: Opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol 2018;15(3, Part B):504–508. doi: 10.1016/j.jacr.2017.12.026.
Humphries T, Si D, Coulter S, Simms M, Xing R. Comparison of deep learning approaches to low dose CT using low intensity and sparse view data. Proc SPIE 2019;10948. doi: https://doi.org/10.1117/12.2512597.
Kim H, Park CM, Lee M, Park SJ, Song YS, Lee J, et al. Impact of Reconstruction Algorithms on CT Radiomic Features of Pulmonary Tumors: Analysis of Intra- and Inter-Reader Variability and Inter-Reconstruction Algorithm Variability. PLoS One 2016;11. doi: https://doi.org/10.1371/journal.pone.0164924.
Greffier J, Hamard A, Pereira F, Barrau C, Pasquier H, et al. Image quality and dose reduction opportunity of deep learning image reconstruction algorithm for CT: a phantom study. Eur Radiol 2020;30:3951–3959. doi: https://doi.org/10.1007/s00330-020-06724-w.
Low-Dose Abdominal CT Using a Deep Learning-Based Denoising Algorithm: A Comparison with CT Reconstructed with Filtered Back Projection or Iterative Reconstruction Algorithm.
Samei E, Richard S. Assessment of the dose reduction potential of a model-based iterative reconstruction algorithm using a task-based performance metrology. Med Phys 2015;42. doi: https://doi.org/10.1118/1.4903899.
Deep learning-based reconstruction in ultra-high-resolution computed tomography: Can image noise caused by high definition detector and the miniaturization of matrix element size be improved?.
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing. p. 234–241, ISBN 978-3-319-24574-4; 2015.
Yu S, Park B., Jeong J. Deep iterative down-up cnn for image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2019.
Yuan H, Jia J, Zhu Z. Sipid: A deep learning framework for sinogram interpolation and image denoising in low-dose ct reconstruction. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 2018, p. 1521–1524. doi: https://doi.org/10.1109/ISBI.2018.8363862.
Bao L, Yang Z, Wang S, Bai D, Lee J. Real image denoising based on multi-scale residual dense block and cascaded u-net with block-connection. In: Proc. of the IEEE/CVF Conf. CVPR. 2020.
Quattoni A, Collins M, Darrell T. Transfer learning for image classification with sparse prototype representations. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, p. 1–8. doi: 10.1109/CVPR.2008.4587637.
Buchholz T, Prakash M, Schmidt D, Krull A, Jug F. Denoiseg: Joint denoising and segmentation. In: Computer Vision - ECCV 2020 Workshops; vol. 12535. 2020, p. 143–146. doi: 10.1007/978-3-030-66415-2_21.
Hajian-Tilaki K. Receiver operating characteristic (roc) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med 2013;4(2):627–635. doi: PMID:24009950; PMCID: PMC3755824.
Midya A, Chakraborty J, Gönen M, Do RKG, Simpson AL. Influence of CT acquisition and reconstruction parameters on radiomic feature reproducibility. J of Med Imaging 2018;011020. doi: https://doi.org/10.1117/1.JMI.5.1.011020.
Influence of sinogram affirmed iterative reconstruction of ct data on image noise characteristics and low-contrast detectability: An objective approach.
Cogswell M, Ahmed F, Girshick BR, Zitnick LC, Batra D. Reducing overfitting in deep networks by decorrelating representations. International conference on learning representations; 2015.
Kanakis M, Bruggemann D, Saha S, Georgoulis S, Obukhov A, Gool LV. Reparameterizing convolutions for incremental multi-task learning without task interference. In: Proc. ECCV 2020; vol. 12365. Springer; 2020, p. 689–707. doi: 10.1007/978-3-030-58565-5_41.
Ponomarenko N, Krivenko S, Egiazarian K, Astola J, Lukin V. Weighted mse based metrics for characterization of visual quality of image denoising methods. 2009.
Liu D, Wen B, Liu X, Huang TS. When image denoising meets high-level vision tasks: A deep learning approach. CoRR 2017;abs/1706.04284. URL http://arxiv.org/abs/1706.04284.
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. TensorFlow: Large-scale machine learning on heterogeneous systems. 2015. doi: http://tensorflow.org/.
Oliver J, Budzevich M, Hunt D, Moros EG, Latifi K., TJ T.J.D., et al. Sensitivity of Image Features to Noise in Conventional and Respiratory-Gated PET/CT Images of Lung Cancer: Uncorrelated Noise Effects. Technol Cancer Res Treat 2017;16(5) 595–608. doi: https://doi.org/10.1177/1533034616661852.
Baker M, Dong F, Primak A, Obuchowski N, Einstein D, Gandhi N, et al. Contrast-to-Noise Ratio and Low-Contrast Object Resolution on Full- and Low-Dose MDCT: Safire Versus Filtered Back Projection in a Low-Contrast Object Phantom and in the Liver. AJR Am J Roentgenol 2012;199. doi: https://doi.org/DOI:10.2214/AJR.11.7421.
Observer Performance in the Detection and Classification of Malignant Hepatic Nodules and Masses with CT Image-Space Denoising and Iterative Reconstruction.
Artmann U, Wueller D. Interaction of image noise, spatial resolution, and low contrast fine detail preservation in digital image processing. In: Digital Photography V; vol. 7250. SPIE; 2009, p. 154–162. doi: 10.1117/12.805927.
Gong H, Hu Q, Walther A, Koo C, Takahashi E, Levin D, et al. Deep-learning-based model observer for a lung nodule detection task in computed tomography. Proc SPIE 2020;042807. doi: https://doi.org/10.1117/1.JMI.7.4.042807.
Reith F, Wandell B. Comparing pattern sensitivity of a convolutional neural network with an ideal observer and support vector machine. arXiv:191105055 2019;.
Alnowami M, Mills G, Awis M, Elangovanr P, Patel M, Halling-Brown M, et al. A deep learning model observer for use in alterative forced choice virtual clinical trials. Proc SPIE 2018;10577. doi: https://doi.org/10.1117/12.2293209.
00111-3&id=gr1.jpg){kind=link}
00111-3&id=gr2.jpg){kind=link}
00111-3&id=gr3.jpg){kind=link}
00111-3&id=gr4.jpg){kind=link}
00111-3&id=gr5.jpg){kind=link}
00111-3&id=gr6.jpg){kind=link}
00111-3&id=gr7.jpg){kind=link}
00111-3&id=gr8.jpg){kind=link}
00111-3&id=gr9.jpg){kind=link}
00111-3&id=gr10.jpg){kind=link}
00111-3&id=gr11.jpg){kind=link}
00111-3&id=gr12.jpg){kind=link}
00111-3&id=gr13.jpg){kind=link}
00111-3&id=gr14.jpg){kind=link}
00111-3&id=gr15.jpg){kind=link}
00111-3&id=gr16.jpg){kind=link}
00111-3&id=gr17.jpg){kind=link}