A high-performance method of deep learning for prostate MR-only radiotherapy planning using an optimized Pix2Pix architecture

Published:October 19, 2022DOI:


      • A competitive method of deep learning (Pix2Pix) for prostate MR-only radiotherapy.
      • Significant variation in image evaluation depending on several parameters.
      • Performances of the Pix2Pix are more accurate than 5 other sCT generation methods.
      • Image and dose errors for the rectum were higher than for other soft tissues.



      The first aim was to generate and compare synthetic-CT (sCT) images using a conditional generative adversarial network (cGAN) method (Pix2Pix) for MRI-only prostate radiotherapy planning by testing several generators, loss functions, and hyper-parameters. The second aim was to compare the optimized Pix2Pix model with five other architectures (bulk-density, atlas-based, patch-based, U-Net, and GAN).


      For 39 patients treated by VMAT for prostate cancer, T2-weighted MRI images were acquired in addition to CT images for treatment planning. sCT images were generated using the Pix2Pix model. The generator, loss function, and hyper-parameters were tuned to improve sCT image generation (in terms of imaging endpoints). The final evaluation was performed by 3-fold cross-validation. This method was compared to five other methods using the following imaging endpoints: the mean absolute error (MAE) and mean error (ME) between sCT and reference CT images (rCT) of the whole pelvis, bones, prostate, bladder, and rectum. For dose planning analysis, the dose-volume histogram metric differences and 3D gamma analysis (local, 1 %/1 mm) were calculated using the sCT and reference CT images.


      Compared with the other architectures, Pix2Pix with Perceptual loss function and generator ResNet 9 blocks showed the lowest MAE (29.5, 107.7, 16.0, 13.4, and 49.1 HU for the whole pelvis, bones, prostate, bladder, and rectum, respectively) and the highest gamma passing rates (99.4 %, using the 1 %/1mm and 10 % dose threshold criterion). Concerning the DVH points, the mean errors were −0.2% for the planning target volume V95%, 0.1 % for the rectum V70Gy, and −0.1 % for the bladder V50Gy.


      The sCT images generated from MRI data with the Pix2Pix architecture had the lowest image errors and similar dose uncertainties (in term of gamma pass-rate and dose-volume histogram metric differences) than other deep learning methods.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Physica Medica: European Journal of Medical Physics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Seco J.
        • Evans P.M.
        Assessing the effect of electron density in photon dose calculations: Effect of electron density in photon dose calculations.
        Med Phys. 2006; 33: 540-552
        • Pathmanathan A.U.
        • McNair H.A.
        • Schmidt M.A.
        • Brand D.H.
        • Delacroix L.
        • Eccles C.L.
        • et al.
        Comparison of prostate delineation on multimodality imaging for MR-guided radiotherapy.
        Br J Radiol. 2019; 92: 20180948
        • Schmidt M.A.
        • Payne G.S.
        Radiotherapy planning using MRI.
        Phys Med Biol. 2015; 60: R323-R361
        • Cusumano D.
        • Boldrini L.
        • Dhont J.
        • Fiorino C.
        • Green O.
        • Güngör G.
        • et al.
        Artificial Intelligence in magnetic Resonance guided Radiotherapy: Medical and physical considerations on state of art and future perspectives.
        Phys Med. 2021; 85: 175-191
        • Kajikawa T.
        • Kadoya N.
        • Tanaka S.
        • Nemoto H.
        • Takahashi N.
        • Chiba T.
        • et al.
        Dose distribution correction for the influence of magnetic field using a deep convolutional neural network for online MR-guided adaptive radiotherapy.
        Phys Med. 2020; 80: 186-192
        • Lambert J.
        • Greer P.B.
        • Menk F.
        • Patterson J.
        • Parker J.
        • Dahl K.
        • et al.
        MRI-guided prostate radiation therapy planning: Investigation of dosimetric accuracy of MRI-based dose planning.
        Radiother Oncol. 2011; 98: 330-334
        • Chen L.
        • Price R.A.
        • Wang L.
        • Li J.
        • Qin L.
        • McNeeley S.
        • et al.
        MRI-based treatment planning for radiotherapy: Dosimetric verification for prostate IMRT.
        Int J Radiat Oncol. 2004; 60: 636-647
        • Lee Y.
        Radiotherapy treatment planning of prostate cancer using magnetic resonance imaging alone.
        Radiother Oncol. 2003; 66: 203-216
        • Largent A.
        • Barateau A.
        • Nunes J.-C.
        • Lafond C.
        • Greer P.B.
        • Dowling J.A.
        • et al.
        Pseudo-CT generation for MRI-only radiation therapy treatment planning: comparison among patch-based, atlas-based, and bulk density methods.
        Int J Radiat Oncol. 2019; 103: 479-490
        • Edmund J.M.
        • Nyholm T.
        A review of substitute CT generation for MRI-only radiation therapy.
        Radiat Oncol. 2017; 12: 28
        • Sjölund J.
        • Forsberg D.
        • Andersson M.
        • Knutsson H.
        Generating patient specific pseudo-CT of the head from MR using atlas-based regression.
        Phys Med Biol. 2015; 60: 825-839
        • Dowling J.A.
        • Sun J.
        • Pichler P.
        • Rivest-Hénault D.
        • Ghose S.
        • Richardson H.
        • et al.
        Automatic substitute computed tomography generation and contouring for magnetic resonance imaging (MRI)-alone external beam radiation therapy from standard MRI sequences.
        Int J Radiat Oncol. 2015; 93: 1144-1153
        • Greer P.B.
        • Dowling J.A.
        • Lambert J.A.
        • Fripp J.
        • Parker J.
        • Denham J.W.
        • et al.
        A magnetic resonance imaging-based workflow for planning radiation therapy for prostate cancer.
        Med J Aust. 2011; 194
      1. Uh J, Merchant TE, Li Y, Li X, Hua C. MRI-based treatment planning with pseudo CT generated through atlas registration: MRI-based treatment planning with atlas approach. Med Phys 2014;41:051711.

        • Siversson C.
        • Nordström F.
        • Nilsson T.
        • Nyholm T.
        • Jonsson J.
        • Gunnlaugsson A.
        • et al.
        Technical Note: MRI only prostate radiotherapy planning using the statistical decomposition algorithm: MRI-only radiotherapy planning using the statistical decomposition algorithm.
        Med Phys. 2015; 42: 6090-6097
        • Demol B.
        • Boydev C.
        • Korhonen J.
        • Reynaert N.
        Dosimetric characterization of MRI-only treatment planning for brain tumors in atlas-based pseudo-CT images generated from standard T 1-weighted MR images: MRI-only treatment planning in atlas-based pseudo-CT images.
        Med Phys. 2016; 43: 6557-6568
        • Andreasen D.
        • Van Leemput K.
        • Edmund J.M.
        A patch-based pseudo-CT approach for MRI-only radiotherapy in the pelvis: A patch-based pseudo-CT approach.
        Med Phys. 2016; 43: 4742-4752
        • Largent A.
        • Barateau A.
        • Nunes J.-C.
        • Mylona E.
        • Castelli J.
        • Lafond C.
        • et al.
        Comparison of deep learning-based and patch-based methods for pseudo-CT generation in MRI-based prostate dose planning.
        Int J Radiat Oncol. 2019; 105: 1137-1150
        • Boulanger M.
        • Nunes J.-C.
        • Chourak H.
        • Largent A.
        • Tahri S.
        • Acosta O.
        • et al.
        Deep learning methods to generate synthetic CT from MRI in radiotherapy: A literature review.
        Phys Med. 2021; 89: 265-281
        • Spadea M.F.
        • Maspero M.
        • Zaffino P.
        • Seco J.
        Deep learning based synthetic-CT generation in radiotherapy and PET: A review.
        Med Phys. 2021; 48: 6537-6566
        • LeCun Y.
        • Bengio Y.
        • Hinton G.
        Deep learning.
        Nature. 2015; 521: 436-444
        • Sahiner B.
        • Pezeshk A.
        • Hadjiiski L.M.
        • Wang X.
        • Drukker K.
        • Cha K.H.
        • et al.
        Deep learning in medical imaging and radiation therapy.
        Med Phys. 2019; 46: e1-e36
        • Jarrett D.
        • Stride E.
        • Vallis K.
        • Gooding M.J.
        Applications and limitations of machine learning in radiation oncology.
        Br J Radiol. 2019; 92: 20190001
        • Meyer P.
        • Noblet V.
        • Mazzara C.
        • Lallement A.
        Survey on deep learning for radiotherapy.
        Comput Biol Med. 2018; 98: 126-146
        • Feng M.
        • Valdes G.
        • Dixit N.
        • Solberg T.D.
        Machine learning in radiation oncology: opportunities, requirements, and needs.
        Front Oncol. 2018; 8: 110
        • Papadimitroulas P.
        • Brocki L.
        • Christopher Chung N.
        • Marchadour W.
        • Vermet F.
        • Gaubert L.
        • et al.
        Artificial intelligence: Deep learning in oncological radiomics and challenges of interpretability and data harmonization.
        Phys Med. 2021; 83: 108-121
        • Barragán-Montero A.
        • Javaid U.
        • Valdés G.
        • Nguyen D.
        • Desbordes P.
        • Macq B.
        • et al.
        Artificial intelligence and machine learning for medical imaging: A technology review.
        Phys Med. 2021; 83: 242-256
        • Arabi H.
        • Dowling J.A.
        • Burgos N.
        • Han X.
        • Greer P.B.
        • Koutsouvelis N.
        • et al.
        Comparative study of algorithms for synthetic CT generation from MRI: consequences for MRI-guided radiation planning in the pelvic region.
        Med Phys. 2018; 45: 5218-5233
        • Neppl S.
        • Landry G.
        • Kurz C.
        • Hansen D.C.
        • Hoyle B.
        • Stöcklein S.
        • et al.
        Evaluation of proton and photon dose distributions recalculated on 2D and 3D Unet-generated pseudoCTs from T1-weighted MR head scans.
        Acta Oncol. 2019; 58: 1429-1434
        • Fu J.
        • Yang Y.
        • Singhrao K.
        • Ruan D.
        • Chu F.-I.
        • Low D.A.
        • et al.
        Deep learning approaches using 2D and 3D convolutional neural networks for generating male pelvic synthetic computed tomography from magnetic resonance imaging.
        Med Phys. 2019; 46: 3788-3798
        • Dinkla A.M.
        • Florkow M.C.
        • Maspero M.
        • Savenije M.H.F.
        • Zijlstra F.
        • Doornaert P.A.H.
        • et al.
        Dosimetric evaluation of synthetic CT for head and neck radiotherapy generated by a patch-based three-dimensional convolutional neural network.
        Med Phys. 2019; 46: 4095-4104
        • Florkow M.C.
        • Zijlstra F.
        • Willemsen K.
        • Maspero M.
        • Berg C.A.T.
        • Kerkmeijer L.G.W.
        • et al.
        Deep learning–based MR-to-CT synthesis: The influence of varying gradient echo–based MR images as input channels.
        Magn Reson Med. 2020; 83: 1429-1441
      2. Thummerer A, de Jong BA, Zaffino P, Meijers A, Marmitt GG, Seco J, et al. Comparison of the suitability of CBCT- and MR-based synthetic CTs for daily adaptive proton therapy in head and neck patients. Phys Med Biol 2020;65:235036.

      3. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. ArXiv150504597 Cs 2015.

      4. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative Adversarial Networks. ArXiv14062661 Cs Stat 2014.

        • Yi X.
        • Walia E.
        • Babyn P.
        Generative adversarial network in medical imaging: A review.
        Med Image Anal. 2019; 58101552
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition.
        in: 2016 IEEE Conf. Comput. Vis. Pattern Recognit. CVPR. IEEE, Las Vegas, NV, USA2016: 770-778
        • Liu X.
        • Emami H.
        • Nejad‐Davarani S.P.
        • Morris E.
        • Schultz L.
        • Dong M.
        • et al.
        Performance of deep learning synthetic CTs for MR-only brain radiation therapy.
        J Appl Clin Med Phys. 2021; 22: 308-317
      5. Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks. ArXiv161107004 Cs 2018.

      6. Ibanez L, Schroeder W. ITK Software Guide n.d.:836.

        • Tustison N.J.
        • Avants B.B.
        • Cook P.A.
        • Yuanjie Zheng
        • Egan A.
        • Yushkevich P.A.
        • et al.
        N4ITK: improved N3 bias correction.
        IEEE Trans Med Imaging. 2010; 29: 1310-1320
      7. Weickert J. Anisotropic Diffusion in Image Processing n.d.:184.

        • Rivest-Hénault D.
        • Dowson N.
        • Greer P.B.
        • Fripp J.
        • Dowling J.A.
        Robust inverse-consistent affine CT–MR registration in MRI-assisted and MRI-alone prostate radiation therapy.
        Med Image Anal. 2015; 23: 56-69
        • Beckendorf V.
        • Guerif S.
        • Le Prisé E.
        • Cosset J.-M.
        • Bougnoux A.
        • Chauvet B.
        • et al.
        70 Gy Versus 80 Gy in localized prostate cancer: 5-year results of GETUG 06 randomized trial.
        Int J Radiat Oncol. 2011; 80: 1056-1063
      8. Maspero M, Savenije MHF, Dinkla AM, Seevinck PR, Intven MPW, Jurgenliemk-Schulz IM, et al. Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy. Phys Med Biol 2018;63:185001.

        • Mylona E.
        • Acosta O.
        • Lizee T.
        • Lafond C.
        • Crehange G.
        • Magné N.
        • et al.
        Voxel-based analysis for identification of urethrovesical subregions predicting urinary toxicity after prostate cancer radiation therapy.
        Int J Radiat Oncol. 2019; 104: 343-354
        • Bahrami A.
        • Karimian A.
        • Arabi H.
        Comparison of different deep learning architectures for synthetic CT generation from MR images.
        Phys Med. 2021; 90: 99-107
        • Bird D.
        • Nix M.G.
        • McCallum H.
        • Teo M.
        • Gilbert A.
        • Casanova N.
        • et al.
        Multicentre, deep learning, synthetic-CT generation for ano-rectal MR-only radiotherapy treatment planning.
        Radiother Oncol. 2021; 156: 23-28
      9. Brou Boni KND, Klein J, Vanquin L, Wagner A, Lacornerie T, Pasquier D, et al. MR to CT synthesis with multicenter data in the pelvic area using a conditional generative adversarial network. Phys Med Biol 2020;65:075002.

      10. Fetty L, Löfstedt T, Heilemann G, Furtado H, Nesvacil N, Nyholm T, et al. Investigating conditional GAN performance with different generator architectures, an ensemble model, and different MR scanners for MR-sCT conversion. Phys Med Biol 2020;65:105004.

        • Cusumano D.
        • Lenkowicz J.
        • Votta C.
        • Boldrini L.
        • Placidi L.
        • Catucci F.
        • et al.
        A deep learning approach to generate synthetic CT in low field MR-guided adaptive radiotherapy for abdominal and pelvic cases.
        Radiother Oncol. 2020; 153: 205-212