If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht 6229 ET, Netherlands.
Affiliations
Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht 6229 ET, Netherlands
Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht 6229 ET, Netherlands
Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht 6229 ET, Netherlands
Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht 6229 ET, Netherlands
Generative models can improve radiomics performance in different tasks when radiomics extracted from low dose CTs.
•
Simulation paired low-high dose CTs trained generative models can be used to denoise low dose CT without re-training.
•
Generative models can improve AUC by 0.05 of radiomics in survival predication and lung cancer diagnosis.
•
Denoising using generative models seems to be a necessary pre-processing step for radiomic features from low dose CTs.
Abstract
Purpose
Radiomics is an active area of research focusing on high throughput feature extraction from medical images with a wide array of applications in clinical practice, such as clinical decision support in oncology. However, noise in low dose computed tomography (CT) scans can impair the accurate extraction of radiomic features. In this article, we investigate the possibility of using deep learning generative models to improve the performance of radiomics from low dose CTs.
Methods
We used two datasets of low dose CT scans – NSCLC Radiogenomics and LIDC-IDRI – as test datasets for two tasks – pre-treatment survival prediction and lung cancer diagnosis. We used encoder-decoder networks and conditional generative adversarial networks (CGANs) trained in a previous study as generative models to transform low dose CT images into full dose CT images. Radiomic features extracted from the original and improved CT scans were used to build two classifiers – a support vector machine (SVM) and a deep attention based multiple instance learning model – for survival prediction and lung cancer diagnosis respectively. Finally, we compared the performance of the models derived from the original and improved CT scans.
Results
Denoising with the encoder-decoder network and the CGAN improved the area under the curve (AUC) of survival prediction from 0.52 to 0.57 (p-value < 0.01). On the other hand, the encoder-decoder network and the CGAN improved the AUC of lung cancer diagnosis from 0.84 to 0.88 and 0.89 respectively (p-value < 0.01). Finally, there are no statistically significant improvements in AUC using encoder-decoder networks and CGAN (p-value = 0.34) when networks trained at 75 and 100 epochs.
Conclusion
Generative models can improve the performance of low dose CT-based radiomics in different tasks. Hence, denoising using generative models seems to be a necessary pre-processing step for calculating radiomic features from low dose CTs.
Reliability of PET/CT shape and heterogeneity features in functional and morphologic components of non–small cell lung cancer tumors: a repeatability analysis in a prospective multicenter cohort.
Reliability of PET/CT shape and heterogeneity features in functional and morphologic components of non–small cell lung cancer tumors: a repeatability analysis in a prospective multicenter cohort.
Comes, Maria Colomba, Daniele La Forgia, Vittorio Didonna, Annarita Fanizzi, Francesco Giotta, Agnese Latorre, Eugenio Martinelli et al., Early prediction of breast cancer recurrence for patients treated with neoadjuvant chemotherapy: a transfer learning approach on DCE-MRIs. Cancers 13, no. 10 (2021): 2298. https://doi.org/10.3390/cancers13102298.
La Forgia, Daniele, Angela Vestito, Maurilia Lasciarrea, Maria Colomba Comes, Sergio Diotaiuti, Francesco Giotta, Agnese Latorre et al., Response predictivity to neoadjuvant therapies in breast cancer: A qualitative analysis of background parenchymal enhancement in DCE-MRI. J. Pers. Med 11, no. 4 (2021): 256. https://doi.org/10.3390/jpm11040256.
Musolino, Stephen V., Joseph DeFranco, and Richard Schlueck. “The ALARA principle in the context of a radiological or nuclear emergency.” Health Phys. 94 (2) (2008): 109–111. https://doi.org/10.1097/01.HP.0000285801.87304.3f.
Bi, Wenya Linda, Ahmed Hosny, Matthew B. Schabath, Maryellen L. Giger, Nicolai J. Birkbak, Alireza Mehrtash, Tavis Allison et al., Artificial intelligence in cancer imaging: clinical challenges and applications. Ca-Cancer J. Clin 69, no. 2 (2019): 127–157. https://doi.org/10.3322/caac.21552.
]. As a tradeoff of low radiation exposure, low dose CTs’ image quality is inferior to that of full dose CTs’, due to the higher noise levels present in low dose CTs. Radiomics applied to low dose CT has already been shown to improve the accuracy of pulmonary nodules analysis for early detection during lung cancer screening [
]. In other words, radiomic features extracted from low dose CTs have lower reliability than the counterparts extracted from full dose CTs. Therefore, prediction models or computer aided diagnosis systems based on radiomic features from low dose CTs will likely be less robust and accurate than those based on radiomic features from full dose CT. Improving the performance of radiomics calculated from low dose CT in different tasks and datasets is therefore a timely and potentially impactful research topic.
] and to recalculate the radiomic features based on the denoised CT. The aim of this article, is to answer the question: should we regard denoising as a preprocessing step for radiomic feature extraction from low dose CT? Image denoising can be regarded as a special case of domain adaptation [
Kollem, Sreedhar, Katta Rama Linga Reddy, and Duggirala Srinivasa Rao. “A review of image denoising and segmentation methods based on medical images.” Int. J. Mach. Learn. Comput. 9, (3) (2019): 288–295. https://doi.org/10.18178/ijmlc.2019.9.3.800.
Parmar, Chintan, Ralph TH Leijenaar, Patrick Grossmann, Emmanuel Rios Velazquez, Johan Bussink, Derek Rietveld, Michelle M. Rietbergen, Benjamin Haibe-Kains, Philippe Lambin, and Hugo JWL Aerts. Radiomic feature clusters and prognostic signatures specific for lung and head & neck cancer. Sci. Rep. 5, no. 1 (2015): 1–10. https://doi.org/10.1038/srep11044.
In addition, we will explore whether more reliable radiomic features result in models with better performance using two real applications of radiomics: pre-treatment survival prediction [
Chen, Junhua, Zeng, Haiyan, Zhang, Cong, et al. “Lung cancer diagnosis using deep attention based multiple instance learning and radiomics.” Med. Phys.. 2022; 00: 00- 00. https://doi.org/10.1002/mp.15539.
] as the MIL problem solver for the sake of interpretability. The authors reported a mean precision of 0.807 with a standard error of the mean (SEM) of 0.069, a recall of 0.870 (SEM 0.061), and an area under the curve (AUC) of 0.842 (SEM 0.074) by using this method.
] – using full dose CTs and simulated paired high-noise low dose CTs. Finally, they showed that radiomic features extracted from low dose CT scans (low-noise CT and high-noise CT) denoised by the models had improved reproducibility. The main differences between [
] and this article is that: 1) we use pre-trained generative models; 2) we use real (not simulated) low dose CTs; and 3) we focus on the improvement in radiomics-based model performance instead of feature reproducibility.
Institutional Review Board approval was not applicable for this study, since the primary source of data was an open access collection on The Cancer Imaging Archive (National Institutes of Health) [
] and all patients’ personal information had been removed from CT scans. This dataset has been used for this study in accordance with the Creative Commons Attribution-NonCommercial 3.0 Unported (CC BY-NC) conditions. The flowchart in Fig. 1 summarizes our study methodology.
], we selected two generative models – encoder-decoder networks and CGANs – that achieved good performance in improving radiomics reproducibility as the experimental models for this study. Moreover, we took the same architecture of encoder-decoder network and CGANs presented in [
Training of encoder-decoder networks and CGANs requires paired low dose and full dose versions of the same CT scan. Although there is an open access dataset containing this kind of scans [
], the exposure of low dose CT scans in the dataset is higher – 50 mA-seconds (mAs) - than in many low dose CT scanning situations. For example, CT scans in the non-small cell lung cancer (NSCLC) Radiogenomics dataset were scanned from 1 to 400 mAs [
As mentioned in the Introduction, we will apply pretrained generative models to improve the performance of low CT radiomics-based models in two tasks: pre-treatment survival prediction and lung cancer diagnosis. For this purpose, we chose the NSCLC Radiogenomics dataset [
The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT Scans: The LIDC/IDRI thoracic CT database of lung nodules.
], because they contain the necessary mask of the region of interest (ROI) for calculating the radiomics features and the images were scanned with low radiation exposure.
NSCLC Radiogenomics is a unique radiogenomic dataset from a cohort of 211 patients with NSCLC [
], from which we used low dose CT images, their respective segmentation masks and clinical data for survival prediction. The lung image database consortium and image database resource initiative (LIDC-IDRI) dataset contains 1018 clinical chest CT scans, along with 157 patients’ diagnoses. We used the diagnoses and their respective CT scans for the lung cancer diagnosis task. Finally, 106 samples of the NSCLC Radiogenomics were selected for survival prediction and 110 samples from LIDC-IDRI for lung cancer diagnosis. The index of selected samples for further investigation can be found in Supplementary Tables 1 and 2. The average radiation exposure of selected samples was mAs (±=SEM) in NSCLC Radiogenomics and mAs in LIDC-IDRI. The distributions of radiation exposure for the two datasets are shown in Supplementary Fig. 1.
Extraction of radiomic features
Before extracting radiomic features from CT images, Hounsfield Unit (HU) value range of CT images were normalized at first. In other words, HU value of pixel in CT images larger than 1000 was set as 1000, and then send the images to extract features.
The masks of the ROIs (tumors) are stored in DICOM format in NSCLC Radiogenomics whilst the segmentation of each nodule is stored in XML file in the LIDC-IDRI dataset. The 3D masks for corresponding ROIs (tumors or nodules) were reconstructed from their corresponding files. We used pyradiomics [
] (version 2.2.0) to calculate 103 radiomic features for further analysis. All features included in the analyses are listed in the Supplementary Table 3.
Radiomics based models’ development
One of the main tasks in the seminal article on radiomics by Aerts et al. [
] with Radial Basis Function (RBF) Kernel as our classifier. SVMs use regularization to prevent overfitting when the number of input variables is high [
Chen, Junhua, Zeng, Haiyan, Zhang, Cong, et al. “Lung cancer diagnosis using deep attention based multiple instance learning and radiomics.” Med. Phys.. 2022; 00: 00- 00. https://doi.org/10.1002/mp.15539.
]. The main characteristic of this classifier is that it can classify groups of samples (e.g. issue a diagnosis based on a set of CT scans from a patient) and reveal the importance of each sample in determining the diagnosis. The architecture of the method is shown in Supplementary Fig. 2. The inputs of the model are the radiomic features and the clinical diagnosis (cancer or not) is the output.
Experiments
We applied the trained generative models to denoise real low dose CT images before extracting the radiomic features. Subsequently, we trained the classification models for survival prediction and lung cancer diagnosis using radiomic features and we compared their performance with that of models trained using radiomic features extracted from low dose CT images.
All denoising experiments for low dose CT images were executed on a Core i7 8565 U CPU with 8 GB of RAM based on pre-trained generative models. Based on training specifications described in [
], generative models were trained 25, 50, 75 and 100 epochs. All four trained models were used for denoising. For internal validation, 40 trials of nested cross validation [
] of RBF kernel SVM were executed and the number of GroupKFold in each trial was set as 5 for survival prediction validation. We adopted the minority oversampling strategy described in [
] for lung cancer diagnosis task to improve the model’s performance due to our dataset being small and imbalanced.
We assessed the models’ performance calculating their area under the receiver-operating characteristics curve (AUC), accuracy and recall (using a probability threshold of 0.5). Finally, we used Student’s t-test, after testing the data for normality, to assess the statistical significance of the differences in model performance results.
Results
An example of an original CT image from the NSCLC Radiogenomics dataset and its denoised counterparts are shown in Fig. 2.
Fig. 2Example of low dose CT denoising: (a) original CT Image from NSCLC Radiogenomics (R01-003, radiation exposure: 7 mAs); (b) image denoised by the CGAN (100 epochs); (c) image denoised by the encoder-decoder network (100 epochs); (d) zoomed region of interests (ROI) of (a); (e) zoomed ROI of (b); and (f) zoomed ROI of (c).
The 4-year survival prediction model based on radiomic features extracted from low dose CTs achieved an AUC of 0.524 with a standard error of the mean (SEM) of 0.042. On the other hand, the survival prediction models based on radiomic features extracted from denoised low dose CTs achieved AUC ranging between 0.54 and 0.58. As shown in Table 1 and Fig. 3, encoder-decoder networks and CGANs can improve radiomics-based models’ performance significantly. The difference between encoder-decoder network and CGAN was not significant when trained for 75 epochs and 100 epochs, similar to what was reported in reference [
Chen, Junhua, Zeng, Haiyan, Zhang, Cong, et al. “Lung cancer diagnosis using deep attention based multiple instance learning and radiomics.” Med. Phys.. 2022; 00: 00- 00. https://doi.org/10.1002/mp.15539.
], our method can achieve an AUC of 0.842 (SEM 0.074) based on radiomic features extracted from the original low dose CT scans from the LIDC-IDRI dataset. The AUCs of the classification models based on radiomics extracted from denoised images range between 0.84 and 0.89 as shown in Table 2 and Fig. 4(c). Models built using radiomic features calculated from denoised images outperformed models developed from the original radiomic features in most experiments. Similarly to survival prediction, the difference between encoder-decoder network and CGAN was not significant when trained for 75 and 100 epochs.
Table 2The AUCs of different models for lung cancer diagnosis.
Fig. 4 (a) and (b) and Table 3 show that denoising had a negative impact in the accuracy and recall of the lung cancer diagnosis classification models, when using a threshold of 0.5.
Table 3Accuracy and recall for lung cancer diagnosis.
In this study, we aimed to assess the potential of generative models to improve the performance of prediction models based on radiomic features extracted from low dose CT scans. The results show that encoder-decoder networks and CGANs can improve the AUC of radiomics for survival prediction and lung cancer diagnosis based on different low dose CT datasets. These findings imply that denoising low dose CT scans using generative models is a convenient pre-processing step before calculating radiomic features to train a predictive or diagnostic model.
The results also show that denoising using generative models might lead to a decrease in accuracy and recall. This might be caused by a shift in the receiver operating characteristic (ROC) curve as a result of the denoising. However, a higher AUC implies that there are other thresholds for which the accuracy and recall are higher with the denoised images. The threshold will differ for each possible application of these models, and a model with a higher AUC will be more likely to have a better accuracy/recall combination.
Another interesting aspect of the results is the variability of the models’ AUCs for different numbers of training epochs. As shown in Fig. 3 and Fig. 4 (c), the performance of the models improves after the first epochs, then deteriorates when training for a higher number of epochs, and finally it seems to improve again after a particular number of training epochs. This tendency seems more significant in Fig. 4 (c) than Fig. 3. This might be explained by a phenomenon that has attracted considerable attention in the deep learning research domain in last few years -- deep double descent [
]. Unfortunately, the mechanisms of this phenomenon are still unclear, and more research on this topic is needed.
It is worth delving into the cause for the observed improvement using generative models. As mentioned previously, we think this improvement is brought on by the denoising effect of generative models to low dose CT. However, as shown in Supplementary Fig. 1 (b), 40% CT images in the LIDC-IDRI dataset were not noisy (since they were scanned with over 200 mAs). Denoising these images using generative models would decrease images’ quality. Therefore, there must be another source of improvement. Our hypothesis for this alternative source of improvement is dose normalization. In other words, generative models not only improved image quality of low dose CT images in dataset but also transfer the imaging exposure of the whole dataset from a wide range to a more compact but unknown range.
One potential limitation of our study is the low AUCs achieved by the models for pre-treatment survival prediction for lung cancer based on radiomic features. However, these are in line with results reported elsewhere. For example, Isensee et al. [
] reported an integrated AUC (iAUC) of 0.620 [95% CI: 0.501–0.756] in TCGA/TCIA dataset using random survival forest to derive a prediction model; Finally, Bae et al. [
] reported an iAUC of 0.590 [95% CI: 0.502, 0.689] for overall survival prediction in Glioblastoma using MRI radiomic features. These relatively low AUCs can be partly explained by the difficulty of pre-treatment survival prediction, especially over a long term (over 2 years). In addition to the information available in the medical image, many other factors can affect survival. In fact, some researchers claim that any AUC over 0.80 is suspect [
]. As a system of hand-crafted features with higher interpretability but lower information representation ability (compared with deep features), it is not surprising that radiomics has a relatively poor performance in survival prediction. Some studies have proposed techniques to improve the performance of radiomics in survival prediction, such as Jia et al. [
] who combined radiomics with deep features to improve the C-index from 0.68 to 0.72. However, these improved results are still low compared to those achieved in diagnosis, and even further developments might still drive the performance up, the performance in survival will remain relatively low due to inherent uncertainty.
Regarding future work, we believe generative models should be trained to keep more information from the original domain. More specific, low level domain adaptation such as denoising for medical images should focus on keeping content information from original domain in the target domain. For example, by adding a content loss term in the cost function, adjusting generative models training method as shown in [
]. Second, more generative models with different architectures should be considered as the test models to find better models for this task. Thirdly, given the important fluctuations in performance across different numbers of training epochs, it is not possible to provide an optimal number of epochs based on our experiments. For consistency, we reported the results from the model trained model for 100 epochs as our final results. However, more studies about the optimal number of epochs are needed in the future. Finally, since the validity of the results of this study are limited to our selected datasets and tasks, further application to more datasets and tasks could reinforce or disprove our findings.
Conclusion
In this study, we assessed the potential of generative models (CGANs and encoder-decoder networks) to improve the performance of low dose CT scan radiomics-based models in two tasks – survival prediction and lung cancer diagnosis – and two datasets – NSCLC Radiogenomics and LIDC-IDRI. SVM and deep attention based MIL were used classifiers in survival prediction and lung cancer diagnosis respectively. The results support the hypothesis that generative models can improve radiomics performance in different tasks and datasets. In conclusion, denoising using generative models is an effective pre-processing step for calculating radiomic features from low dose CT.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
JC is supported by a China Scholarship Council scholarship (201906540036). The remaining authors acknowledge funding support from the following: STRaTegy (STW 14930), BIONIC (NWO 629.002.205), TRAIN (NWO 629.002.212), CARRIER (NWO 628.011.212) and a personal research grant by The Hanarth Funds Foundation for LW.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
Reliability of PET/CT shape and heterogeneity features in functional and morphologic components of non–small cell lung cancer tumors: a repeatability analysis in a prospective multicenter cohort.
Comes, Maria Colomba, Daniele La Forgia, Vittorio Didonna, Annarita Fanizzi, Francesco Giotta, Agnese Latorre, Eugenio Martinelli et al., Early prediction of breast cancer recurrence for patients treated with neoadjuvant chemotherapy: a transfer learning approach on DCE-MRIs. Cancers 13, no. 10 (2021): 2298. https://doi.org/10.3390/cancers13102298.
La Forgia, Daniele, Angela Vestito, Maurilia Lasciarrea, Maria Colomba Comes, Sergio Diotaiuti, Francesco Giotta, Agnese Latorre et al., Response predictivity to neoadjuvant therapies in breast cancer: A qualitative analysis of background parenchymal enhancement in DCE-MRI. J. Pers. Med 11, no. 4 (2021): 256. https://doi.org/10.3390/jpm11040256.
Musolino, Stephen V., Joseph DeFranco, and Richard Schlueck. “The ALARA principle in the context of a radiological or nuclear emergency.” Health Phys. 94 (2) (2008): 109–111. https://doi.org/10.1097/01.HP.0000285801.87304.3f.
Bi, Wenya Linda, Ahmed Hosny, Matthew B. Schabath, Maryellen L. Giger, Nicolai J. Birkbak, Alireza Mehrtash, Tavis Allison et al., Artificial intelligence in cancer imaging: clinical challenges and applications. Ca-Cancer J. Clin 69, no. 2 (2019): 127–157. https://doi.org/10.3322/caac.21552.
Kollem, Sreedhar, Katta Rama Linga Reddy, and Duggirala Srinivasa Rao. “A review of image denoising and segmentation methods based on medical images.” Int. J. Mach. Learn. Comput. 9, (3) (2019): 288–295. https://doi.org/10.18178/ijmlc.2019.9.3.800.
Parmar, Chintan, Ralph TH Leijenaar, Patrick Grossmann, Emmanuel Rios Velazquez, Johan Bussink, Derek Rietveld, Michelle M. Rietbergen, Benjamin Haibe-Kains, Philippe Lambin, and Hugo JWL Aerts. Radiomic feature clusters and prognostic signatures specific for lung and head & neck cancer. Sci. Rep. 5, no. 1 (2015): 1–10. https://doi.org/10.1038/srep11044.
Chen, Junhua, Zeng, Haiyan, Zhang, Cong, et al. “Lung cancer diagnosis using deep attention based multiple instance learning and radiomics.” Med. Phys.. 2022; 00: 00- 00. https://doi.org/10.1002/mp.15539.
The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT Scans: The LIDC/IDRI thoracic CT database of lung nodules.
01963-9&id=gr1.jpg){kind=link}
01963-9&id=gr2.jpg){kind=link}
01963-9&id=gr3.jpg){kind=link}
01963-9&id=gr4.jpg){kind=link}