Original paper| Volume 55, P98-106, November 2018

Pareto-optimal plans as ground truth for validation of a commercial system for knowledge-based DVH-prediction


      • 115 Pareto-optimal plans were used to assess the prediction accuracy RapidPlan.
      • Prediction models were generated using 20, 30, 45, 55 and 114 plans.
      • Intrinsic accuracy of RapidPlan predictions was estimated for rectum, bladder and anus.
      • For model-114, clinically relevant deviations in DVH predictions were observed.
      • With smaller model size, the prediction showed dependence on the training set.



      Treatment plans manually generated in clinical routine may suffer from variations and inconsistencies in quality. Using such plans for validating a DVH prediction algorithm might obscure its intrinsic prediction accuracy. In this study we used a recently published large database of Pareto-optimal prostate cancer plans to assess the prediction accuracy of a commercial knowledge-based DVH prediction algorithm, RapidPlan. The database plans were consistently generated with automated planning using an independent optimizer, and can be considered as aground truth of plan quality.


      Prediction models were generated using training sets with 20, 30, 45, 55 and 114 Pareto-optimal plans. Model-20 and Model-30 were built using 5 groups of randomly selected training patients. For 60 independent Pareto-optimal validation plans, predicted and database DVHs were compared.


      For model-114, differences between predicted and database mean doses of more than ± 10% in rectum, anus and bladder, occurred for 23.3%, 55.0%, and 6.7% of the validation plans, respectively. For rectum V65Gy and V75Gy, differences outside the ±10% range were observed in 21.7% and 70.0% of validation plans, respectively. For 61.7% of validation plans, inaccuracies in predicted rectum DVHs resulted in a deviation in predicted NTCP for rectal bleeding outside ±10%. With smaller training sets the DVH prediction performance deteriorated, showing dependence on the selected training patients.


      Even when analysed with Pareto-optimal plans with highly consistent quality, clinically relevant deviations in DVH predictions were observed. Such deviations could potentially result in suboptimal plans for new patients. Further research on DVH prediction models is warranted.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Physica Medica: European Journal of Medical Physics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Sharpe M.B.
        • Moore K.L.
        • Orton
        CG:Point/counterpoint: within the next ten years treatment planning will become fully automated without the need for human intervention.
        Med Phys. 2014; 42: 120601-120604
        • Men K.
        • Zhang T.
        • Chen X.
        • Chen B.
        • Tang Y.
        • Wang S.
        • et al.
        Fully automatic and robust segmentation of the clinical target volume for radiotherapy of breast cancer using big data and deep learning.
        Phys Medica. 2018; 50: 13-19
        • Kawata Y.
        • Arimura H.
        • Ikushima K.
        • Jin Z.
        • Morita K.
        • Tokunaga C.
        • Yabu-uchi H.
        • Shioyama Y.
        • et al.
        Impact of pixel-based machine-learning techniques on automated frameworks for delineation of gross tumor volume regions for stereotactic body radiation therapy.
        Phys Medica. 2017; 42 (141–149A)
        • Fogliata A.
        • Belosi F.
        • Clivio A.
        • Navarria P.
        • Nicolini G.
        • Scorsetti M.
        • et al.
        On the pre-clinical validation of a commercial model-based optimisation engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
        Radiother Oncol. 2014; 113: 385-391
        • Hussein M.
        • South C.P.
        • Barry M.A.
        • Adams E.J.
        • Jordan T.J.
        • Stewart A.J.
        • et al.
        Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
        Radiother Oncol. 2016; 120: 473-479
        • Cagni E.
        • Botti A.
        • Micera R.
        • Galeandro M.
        • Sghedoni R.
        • Orlandi M.
        • et al.
        Knowledge-based treatment planning: an inter-technique and inter-system feasibility study for prostate cancer.
        Phys Med. 2017; 36: 38-45
        • Tol J.P.
        • Delaney A.R.
        • Dahele M.
        • Slotman B.J.
        • Verbakel W.F.
        Evaluation of a knowledge-based planning solution for head and neck cancer.
        Int J Radiat Oncol Biol Phys. 2015; 91: 612-620
        • Tol J.
        • Dahele M.
        • Peltola J.
        • Nord J.
        • Slotman B.
        • Verbakel W.F.
        Automatic interactive optimization for volumetric modulated arc therapy planning.
        Radiat Oncol. 2015; 10: 75
        • Fogliata A.
        • Wang P.
        • Belosi F.
        • et al.
        Assessment of a model based optimization engine for volumetric modulated arc therapy for patients with advanced hepatocellular cancer.
        Radiat Oncol. 2014; 9: 236
        • Hansen C.R.
        • Bertelsen A.
        • Hazell I.
        • et al.
        Automatic treatment planning improves the clinical quality of head and neck cancer treatment plans.
        Clin Trans Radiat Oncol. 2016; 1: 2-8
        • Hazell I.
        • Bzdusek K.
        • Kumar P.
        • et al.
        Automatic planning of head and neck treatment plans.
        J Appl Clin Med Phys. 2016; Jan;; 17: 272-282
        • Krayenbuehl J.
        • Norton I.
        • Studer G.
        • Guckenberger M.
        Evaluation of an automated knowledge based treatment planning system for head and neck.
        Radiad Oncol. 2015; 10: 226
        • Breedveld S.
        • Storchi P.
        • Voet P.
        • Heijmen B.
        iCycle: Integrated, multicriterial beam angle, and profile optimization for generation of coplanar and noncoplanar IMRT plans.
        Med Phys. 2012; 39: 951-963
        • Voet P.W.
        • Dirkx M.L.
        • Breedveld S.
        • Fransen D.
        • Levendag P.C.
        • Heijmen B.J.M.
        Toward fully automated multicriterial plan generation: a prospective clinical study.
        Int J Radiat Oncol Biol Phys. 2013; 85: 866-872
        • Sharfo A.W.
        • Breedveld S.
        • Voet P.W.
        • Heijkoop S.T.
        • Mens J.M.
        • Hoogeman M.S.
        • et al.
        Validation of fully automated VMAT plan generation for library-based plan-of-the-day cervical cancer radiotherapy.
        PLoS ONE. 2016; 11 (e0169202)
        • Della Gala G.
        • Dirkx M.L.
        • Hoekstra N.
        • et al.
        Fully automated VMAT treatment planning for advanced-stage NSCLC patients.
        Strahlenther Onkol. 2017;
        • Voet P.W.
        • Dirkx M.L.
        • Breedveld S.
        • Al-Mamgani A.
        • Incrocci L.
        • Heijmen B.J.M.
        Fully automated volumetric modulated arc therapy plan generation for prostate cancer patients.
        Int J Radiat Oncol Biol Phys. 2014; 88: 1175-1179
        • Gallio E.
        • Giglioli F.R.
        • Girardi A.
        • Guarneri A.
        • Ricardi U.
        • Ropolo R.
        • et al.
        Evaluation of a commercial automatic treatment planning system for liver stereotactic body radiation therapy treatments.
        Phys Medica. 2018; 46: 153-159
      1. Wang Y., Petit SF. Radiotherapy treatment planning QA.; 2016.

        • Wang Y.
        • Breedveld S.
        • Heijmen B.J.
        • Petit S.F.
        Evaluation of plan quality assurance models for prostate cancer patients based on fully automatically generated Pareto-optimal treatment plans.
        Phys Med Biol. 2016; 61: 4268
      2. Varian Medical System. Eclipse Photon and Electron reference guide v.13.7, 201 Palo Alto CA, 269: 360; 2015.

      3. Varian Model Analytics; 2018.

        • Tol J.P.
        • Dahele M.
        • Delaney A.R.
        • Slotman B.J.
        • Verbakel W.F.
        Can knowledge-based DVH predictions be used for automated, individualized quality assurance of radiotherapy treatment plans?.
        Radiat Oncol. 2015; 10: 234
        • Lyman J.T.
        Complication probability as assessed from dose–volume histograms.
        Radiat Res Suppl. 1985; 8: S13-S19
        • Landoni V.
        • Fiorino C.
        • Cozzarini C.
        • Sanguineti G.
        • Valdagni R.
        • Rancati T.
        Predicting toxicity in radiotherapy for prostate cancer.
        Physica Med. 2016; 32: 521-532
        • Michalski J.M.
        • Gay H.
        • Jackson A.
        • Tucker S.L.
        • Deasy J.O.
        Radiation dose-volume effects in radiation-induced rectal injury.
        Int J Radiat Oncol Biol Phys. 2016; 76: S123-S129
        • Yorke E.D.
        Modeling the effects of inhomogeneous dose distributions in normal tissues.
        Semin Radiat Oncol. 2001; 11: 197-209
        • Aluwini S.
        • Pos F.
        • Schimmel E.
        • Krol S.
        • et al.
        Hypofractionated versus conventionally fractionated radiotherapy for patients with prostate cancer (HYPRO): late toxicity results from a randomised, non-inferiority, phase 3 trial.
        Lancet Oncol. 2016; 17: 464-474
        • Delaney A.R.
        • Tol J.P.
        • Dahele M.
        • Cuijpers J.
        • Slotman B.J.
        • Verbakel W.F.A.R.
        Effect of dosimetric outliers on the performance of a commercial knowledgebased planning solution.
        Int J Radiat Oncol Biol Phys. 2016; 94: 469-477
        • Fogliata A.
        • Reggiori G.
        • Stravato A.
        • Lobefalo F.
        • Franzese C.
        • Franceschini D.
        • Tomatis S.
        • Mancosu P.
        • Scorsetti M.
        • Cozzi L.
        RapidPlan head and neck model: the objectives and possible clinical benefit.
        Radiat Oncol. 2017; 12: 73
        • Boutilier J.J.
        • Craig T.
        • Sharpe M.B.
        • Chan T.C.
        Sample size requirements for knowledge-based treatment planning.
        Med Phys. 2016; 43: 1212
        • Appenzoller L.M.
        • Michalski J.M.
        • Thorstad W.L.
        • Mutic S.
        • Moore K.L.
        Predicting dose-volume histograms for organs-at-risk in IMRT planning.
        Med Phys. 2012; 39: 7446-7461