Advertisement

Evaluation of DIR algorithm performance in real patients for radiotherapy treatments: A systematic review of operator-dependent strategies

Published:August 22, 2022DOI:https://doi.org/10.1016/j.ejmp.2022.08.011

      Highlights

      • The performance of deformable medical image registration (DIR) is a major concern.
      • We propose a state-of-the-art DIR algorithm performance quantification.
      • We focused only on studies using operator-based methods applied to real patients.
      • This review should help to implement DIR algorithms in clinical practice.

      Abstract

      Purpose

      The performance of deformable medical image registration (DIR) algorithms has become a major concern.

      Methods

      We aimed to obtain updated information on DIR algorithm performance quantification through a literature review of articles published between 2010 and 2022. We focused only on studies using operator-based methods to treat real patients. The PubMed, Google Scholar and Embase databases were searched following PRISMA guidelines.

      Results

      One hundred and seven articles were identified. The mean number of patients and registrations per publication was 20 and 63, respectively. We found 23 different geometric metrics appearing at least twice, and the dosimetric impact of DIR was quantified in 32 articles. Forty-eight different at-risk organs were described, and target volumes were studied in 43 publications. Prostate, head-and-neck and thoracic locations represented more than ¾ of the studied locations. We summarized the type of DIR and the images used, and other key elements. Intra/interobserver variability, threshold values and the correlation between metrics were also discussed.

      Conclusions

      This literature review covers the past decade and should facilitate the implementation of DIR algorithms in clinical practice by providing practical and pertinent information to quantify their performance on real patients.

      Keywords

      Introduction

      The use of deformable medical image registration (DIR) algorithms has been steadily increasing in radiotherapy over the last decade [
      • Yuen J.
      • Barber J.
      • Ralston A.
      • Gray A.
      • Walker A.
      • Hardcastle N.
      • et al.
      An international survey on the clinical use of rigid and deformable image registration in radiotherapy.
      ,
      • Paganelli C.
      • Meschini G.
      • Molinelli S.
      • Riboldi M.
      • Baroni G.
      Patient-specific validation of deformable image registration in radiation therapy: Overview and caveats.
      ]. Today, the applications are numerous, including atlas-based segmentation, multimodal image fusion at planning or treatment, contour propagation, dose accumulation in adaptive radiotherapy (ART) or reirradiation, MRI-based dose calculation, toxicity prediction, etc. [

      Kadoya N, Kito S, Kurooka M, Saito M, Takemura A, Tohyama N, et al. Factual survey of the clinical use of deformable image registration software for radiotherapy in Japan. Journal of Radiation Research 2019;60:546–53. 10.1093/jrr/rrz034.

      ,
      • Rigaud B.
      • Simon A.
      • Castelli J.
      • Lafond C.
      • Acosta O.
      • Haigron P.
      • et al.
      Deformable image registration for radiation therapy: principle, methods, applications and evaluation.
      ]. The characterization of the performance of these DIR algorithms has become a major concern for both developers and clinicians. Numerous articles have been published in which their authors have sought to quantify the performance of DIR algorithms.
      In the context of the increasing use of DIR algorithms in radiotherapy, a quality assurance program was proposed by the AAPM in its report 132 to harmonize quantification methods [
      • Brock K.K.
      • Mutic S.
      • McNutt T.R.
      • Li H.
      • Kessler M.L.
      Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the AAPM Radiation Therapy Committee Task Group No. 132.
      ]. In this report, the authors insisted on the commissioning stage, which consists of validating the performance of DIR algorithms before their use in clinical practice. The following types of tests are recommended during this stage: end-to-end tests on physical phantoms, tests on digital phantoms and, finally, on real patients. The testing methods for phantoms are clearly explained, and precise phantoms are even proposed, allowing a simple and homogeneous application of the recommendations.
      Clinical trials are crucial because they integrate clinical reality’s complexity, including anatomical deformations and image quality that are challenging to reproduce on phantoms [
      • Akbarzadeh A.
      • Gutierrez D.
      • Baskin A.
      • Ay M.R.
      • Ahmadian A.
      • Alam N.R.
      • et al.
      Evaluation of whole-body MR to CT deformable image registration.
      ]. The recommendations in AAPM Report 132 are specific to phantom tests and are less directive for patient-specific tests, mainly because of the lack of ground truth, making them challenging to apply in real-life scenarios [
      • Alam F.
      • Rahman S.U.
      • Khalil A.
      • Khusro S.
      Sajjad M.
      ,
      • Viergever M.A.
      • Maintz J.B.A.
      • Klein S.
      • Murphy K.
      • Staring M.
      • Pluim J.P.W.
      A survey of medical image registration – under review.
      ,
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      ,
      • Rong Y.i.
      • Rosu-Bubulac M.
      • Benedict S.H.
      • Cui Y.
      • Ruo R.
      • Connell T.
      • et al.
      Rigid and Deformable Image Registration for Radiation Therapy: A Self-Study Evaluation Guide for NRG Oncology Clinical Trial Participation.
      ]. In a review of articles published between 2009 and 2017, Paganelli et al. sought to bridge this gap by providing insights into patient-specific DIR validation strategies [
      • Paganelli C.
      • Meschini G.
      • Molinelli S.
      • Riboldi M.
      • Baroni G.
      Patient-specific validation of deformable image registration in radiation therapy: Overview and caveats.
      ]. Guidelines to improve the process are proposed, and a detailed focus on the latest advances in research on this topic making this article an indispensable complement to the AAPM report. Another essential complement was published by the NRG Oncology Medical Physics [
      • Rong Y.i.
      • Rosu-Bubulac M.
      • Benedict S.H.
      • Cui Y.
      • Ruo R.
      • Connell T.
      • et al.
      Rigid and Deformable Image Registration for Radiation Therapy: A Self-Study Evaluation Guide for NRG Oncology Clinical Trial Participation.
      ]. In this guide, additional recommendations are given, particularly in relation to the current DIR system limitations and of their site-specific properties.
      For patient-specific validation of DIR, Paganelli et al. distinguished between operator-dependent and automatic strategies applied for geometric or dosimetric accuracy assessment. Automatic quantification strategies do not require ground truth, overcome operator variability, and decrease the workload. However, they are based on algorithms that are not yet easily available in a clinical environment and whose interpretation is not straightforward [
      • Juan-Cruz C.
      • Fast M.F.
      • Sonke J.-J.
      A multivariable study of deformable image registration evaluation metrics in 4DCT of thoracic cancer patients.
      ].
      In our article, we propose a state-of-the-art DIR algorithm performance quantification through a literature review of articles published between 2010 and 2022. We focused only on studies using operator-based methods applied to real patients. Our objective is to highlight the most used methods and metrics, the anatomical locations, the number of patients and the number of registrations considered sufficient by the different teams to evaluate the performance of a DIR algorithm. We also highlight the evolution of practices until 2022, including the impact of the publication of AAPM TG132 in 2018. The methods for taking into account intra/interobserver variability, threshold values and the correlation between metrics are also summarized. A reader wishing to characterize the performance of a DIR on real patients can then easily find key information to conduct their study.

      Materials and methods

      Selection of reviewed articles

      We followed the recommendations of the PRISMA method to conduct this literature review [
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement.
      ]. More specifically, we followed the guidelines of this method translated into French [
      • Mateo S.
      Procédure pour conduire avec succès une revue de littérature selon la méthode PRISMA.
      ], describing the use of the open-source bibliography software Zotero [].
      Articles published between 2010 and May 2022 were selected with the following keywords in the article title: [[deformable AND registration] OR dir] AND [evaluation OR assessment OR performance OR validation OR assessing OR accuracy OR propagation OR review OR survey OR quantification OR quality] – phantom. A preliminary sorting was performed based on the reading of the abstract alone, which consisted of rejecting articles not related to the topic of our publication. This methodology was applied to the following three databases: Google Scholar, PubMed, and Embase. The results of these three searches were merged, and duplicates were removed.
      A second refined sorting was then performed on this merged base by reading the body of the article. Only papers in which the performance of a DIR algorithm is quantified on real patients and by operator-based methods were retained: we excluded publications in which phantoms, synthetic transformations or public patient databases (such as DIRLAB or POPI) were used. We also excluded proofs-of-concept and publications that did not provide sufficient detail.
      Finally, we added articles from other sources that would have been missed in our selection process, for example (as mentioned earlier), some of those cited by Paganelli et al.[
      • Paganelli C.
      • Meschini G.
      • Molinelli S.
      • Riboldi M.
      • Baroni G.
      Patient-specific validation of deformable image registration in radiation therapy: Overview and caveats.
      ], or the references obtained during reading the articles in a secondary, detailed sorting of the articles.

      Synthesis and analysis of the information collected

      The following items were collected systematically, and the following statistics were calculated:
      • -
        Main localization: head-and-neck (H&N), chest, prostate, abdomen, brain, cervix, whole body
      • -
        Structures/landmarks used for ground truth
      • -
        Number of patients in the database
      • -
        Number of registrations used for DIR performance quantification
      • -
        DIR: mono- or multimodal
      • -
        Type of images: computed tomography (CT), cone-beam CT (CBCT), megavoltage CT (MVCT), magnetic resonance (MR), ultrasound (US)
      • -
        Type of DIR: commercial, in-house or open-source
      • -
        Nature of DIR technique: B-spline, demons, free-form deformation, optical flow, level-set, finite element method, salient feature based, atlas, deep-learning, ANACONDA (Raysearch, Stockholm, Sweden), MORFEUS (Raysearch, Stockholm, Sweden), ADMIRE (Elekta AB, Stockholm, Sweden), MIM hybrid (MIM, OH, USA), DIRART [
        • Yang D.
        • Brame S.
        • El Naqa I.
        • Aditya A.
        • Wu Y.
        • Goddu S.M.
        • et al.
        Technical note: DIRART–A software suite for deformable image registration and adaptive radiotherapy research.
        ], intensity (if the exact nature is not specified or doesn’t belong to other intensity-based methods), hybrid (if the exact nature is not specified or doesn’t belong to other hybrid-based methods) or other (if we did not succeed in classifying it in one of these categories)
      • -
        Type of metrics used to assess DIR performance. “DOSE” is specified when a dosimetric analysis is performed, regardless of the dosimetric metric used.
      • -
        Thresholds: purely descriptive quantitative analysis, or definition of thresholds to validate the DIR (only for geometric analysis)
      • -
        Ground truth: number of observers used to obtain the ground truth (contours or landmarks). “STAPLE” is mentioned if this method is used.
      • -
        Dosimetric analysis: “yes” if the impact of the DIR has been quantified on the dose distribution and “no” if not
      • -
        Correlation among metrics studied in the article: yes or non. If yes, between what type of metrics.

      Results

      One hundred and seven articles were identified between 2010 and May 2022; details of the selection process are given in Fig. 1. The synthesis of the elements found in the articles is given in Table 1.
      Table 1Summary of the 107 articles, sorted by localization and year of publication.
      DatabaseEvaluation
      Article (First author, year, reference)Main localizationStructures/Landmarks#patients#registrationsMono/multimodalDIR: commercial/in-house/open sourceNature of DIRMetricsThresholdsGround-truthDosimetric evaluationCorrelation among metrics
      Brock2010
      • Brock KK, Deformable Registration Accuracy Consortium
      Results of a multi-institution deformable registration accuracy study (MIDRAS).
      Prostate, thorax abdomen91 landmarks34Mono and multiCT/CT, CT/MRCommercial, in-house, open sourceDEM, BSP, otherTRENoOne observerNoNo
      Fallone2010
      • Fallone B.G.
      • Rivest D.R.C.
      • Riauka T.A.
      • Murtha A.D.
      Assessment of a commercially available automatic deformable registration system.
      ProstateProstate55MultiCT/MRCommercialOtherTC, DSC, MI, SCR, MCSNoOne observerNoDSC/MI, MCS/MI, DSC/SCR, MCS/SCR
      Thörnqvist2010
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      ProstateProstate, nodes, rectum, bladder435MonoCT/CTCommercialDEMDSC, SENS, PPV, QRGrading for QROne observerNoDSC/QR, PPV/SENS
      Thor2011
      • Thor M.
      • Petersen J.B.B.
      • Bentzen L.
      • Høyer M.
      • Muren L.P.
      Deformable image registration for contour propagation from CT to cone-beam CT scans in radiotherapy of prostate cancer.
      ProstateProstate, rectum, bladder536MultiCT/CBCTCommercialDEMDSC, QRGrading for QROne observerNoDSC/QR
      Hu2012
      • Hu Y.
      • Ahmed H.U.
      • Taylor Z.
      • Allen C.
      • Emberton M.
      • Hawkes D.
      • et al.
      MR to ultrasound registration for image-guided prostate interventions.
      ProstateLandmarks8800MultiMR/USIn-houseFEMTRENoOne observerNoNo
      Kim2013
      • Kim J.
      • Kumar S.
      • Liu C.
      • Zhong H.
      • Pradhan D.
      • Shah M.
      • et al.
      A novel approach for establishing benchmark CBCT/CT deformable image registrations in prostate cancer radiotherapy.
      Prostate20 landmarks + prostate, bladder, rectum, SV33MultiCT/CBCTOpen sourceBSPDSC, JACO, TRE, VOLNoFive observers contoured twiceNoNo
      Thor2013
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ProstateRectum, bladder978MonoCT/CTCommercialDEMDSC, QR, VOL, DOSEGrading for QROne observerYesDOSE/DSC, DOSE/QR, DOSE/VOL
      Thor2014
      • Thor M.
      • Andersen E.S.
      • Petersen J.B.B.
      • Sørensen T.S.
      • Noe K.Ø.
      • Tanderup K.
      • et al.
      Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy.
      ProstateProstate, bladder978MonoCT/CTCommercial
      Research version.
      , in-house
      DEM, otherDSC, VOL, DOSE, NTCPMedian DSC < 0.9One observerYesDSC/VOL
      Fedoro2015
      • Fedorov A.
      • Khallaghi S.
      • Sánchez C.A.
      • Lasso A.
      • Fels S.
      • Tuncali K.
      • et al.
      Open-source image registration for MRI–TRUS fusion-guided prostate interventions.
      ProstateLandmarks1111MultiMR/USIn-houseBSP, FEMTRENoOne observerNoNo
      Gardner2015
      • Gardner S.J.
      • Wen N.
      • Kim J.
      • Liu C.
      • Pradhan D.
      • Aref I.
      • et al.
      Contouring variability of human- and deformable-generated contours in radiotherapy for prostate cancer.
      ProstateProstate, bladder, rectum1040MonoCBCT/CBCTCommercialBSPDSC, HD, COM, CDNoFive observers and STAPLENoNo
      Moulton2015
      • Moulton C.R.
      • House M.J.
      • Lye V.
      • Tang C.I.
      • Krawiec M.
      • Joseph D.J.
      • et al.
      Registering prostate external beam radiotherapy with a boost from high-dose-rate brachytherapy: a comparative evaluation of deformable registration algorithms.
      ProstateRectum6464MonoCT/CTCommercialBSP, DEM, OF, DIRARTDSC, HD, JACO, ASuD, MI, MSE, QRNoOne observerNoNo
      Mayer2016
      • Mayer A.
      • Zholkover A.
      • Portnoy O.
      • Raviv G.
      • Konen E.
      • Symon Z.
      Deformable registration of trans-rectal ultrasound (TRUS) and magnetic resonance imaging (MRI) for focal prostate brachytherapy.
      ProstateLandmarks1010MultiMR/USIn-houseBSPTRENoOne observerNoNo
      Riegel2016
      • Riegel A.C.
      • Antone J.G.
      • Zhang H.
      • Jain P.
      • Raince J.
      • Rea A.
      • et al.
      Deformable image registration and interobserver variation in contour propagation for radiation therapy planning.
      Prostate, H&NBS, SC, oral cavity, larynx, PG, ON, eyes

      Prostate, bladder, rectum
      5050MonoCT/CTCommercialBSPHDNoTwo observersNoNo
      Saleh2016
      • Saleh Z.
      • Thor M.
      • Apte A.P.
      • Sharp G.
      • Tang X.
      • Veeraraghavan H.
      • et al.
      A multiple-image-based method to evaluate the performance of deformable image registration in the pelvis.
      ProstateBladder, rectum38228MonoCT/CTIn-houseBSPDDM, ICE, TE, DSC, VOLNoOne observerNoDDM/TE, DDM/ICE, TE/ICE, DDM/DSC, DDM/VOL
      Takaya2017
      • Takayama Y.
      • Kadoya N.
      • Yamamoto T.
      • Ito K.
      • Chiba M.
      • Fujiwara K.
      • et al.
      Evaluation of the performance of deformable image registration between planning CT and CBCT images for the pelvic region: comparison between hybrid and intensity-based DIR.
      ProstateProstate, rectum, bladder, SV1095MultiCBCT/CTCommercialAnaconda, intensityDSC, COMNoOne observerNoNo
      Velec2017
      • Velec M.
      • Moseley J.L.
      • Svensson S.
      • Hårdemark B.
      • Jaffray D.A.
      • Brock K.K.
      Validation of biomechanical deformable image registration in the abdomen, thorax, and pelvis in a commercial radiotherapy treatment planning system.
      Prostate, thorax, abdomenLandmarks + liver, spleen, body, stomach, kidneys, lungs, tumors, heart, bronchus, bone, breast, prostate7474Mono and multiCT/CT, CT/MR, MR/MRCommercialMorfeus, anacondaDTA, TRE, DSC, CC, MINoOne observerNoNo
      Woerner2017
      • Woerner A.J.
      • Choi M.
      • Harkenrider M.M.
      • Roeske J.C.
      • Surucu M.
      Evaluation of Deformable Image Registration-Based Contour Propagation From Planning CT to Cone-Beam CT.
      Prostate,H&N, abdomenEsophagus, mandible, PG, SC, prostate, bladder, rectum, SV, pancreas, liver, kidneys1616MultiCT/CBCTCommercialBSPDSC, MSD, HDNoTwo observersNoNo
      Yang2017
      • Yang X.
      • Rossi P.J.
      • Jani A.B.
      • Mao H.
      • Zhou Z.
      • Curran W.J.
      • et al.
      Improved prostate delineation in prostate HDR brachytherapy with TRUS-CT deformable registration technology: A pilot study with MRI validation.
      ProstateProstate1616MultiUS/CTIn-houseOtherDSC, VOLNoThree observers on eight patientsNoNo
      Jamema2018
      • Jamema S.V.
      • Phurailatpam R.
      • Paul S.N.
      • Joshi K.
      • Deshpande D.D.
      Commissioning and validation of commercial deformable image registration software for adaptive contouring.
      Prostate, H&N, brain, cervixPG, TG, mandible, eye, brain, BS, bladder, rectum5555multiCT/CBCTCommercialDEMDSC,HD, COMNoOne observerNoNo
      Moteggi2018
      • Motegi K.
      • Tachibana H.
      • Motegi A.
      • Hotta K.
      • Baba H.
      • Akimoto T.
      Usefulness of hybrid deformable image registration algorithms in prostate radiation therapy.
      ProstateProstate, bladder, rectum, SV1010MultiCT/CBCTCommercialAnaconda, intensity, FFD, MIMHDSC, TRE, JACODSC > 0.8

      TRE < voxel size
      One observerNoNo
      Poulin2018
      • Poulin E.
      • Boudam K.
      • Pinter C.
      • Kadoury S.
      • Lasso A.
      • Fichtinger G.
      • et al.
      Validation of MRI to TRUS registration for high-dose-rate prostate brachytherapy.
      ProstateProstate1515MultiMR/USOpen-sourceBSPHD, DSC, TRE, VOLNoOne observerNoNo
      Saito2018
      • Saito M.
      • Shibata Y.
      • Sano N.
      • Kuriyama K.
      • Komiyama T.
      • Marino K.
      • et al.
      Evaluation of Deformable Image Registration and Dose Accumulation Using Histogram Matching Algorithm between kVCT and MVCT with Helical Tomotherapy.
      ProstateProstate, SV, rectum, bladder535MultiCT/MVCTCommercialIntensityDSC, DOSENoOne or two observersYesNo
      Elmahdy2019
      • Elmahdy M.S.
      • Jagt T.
      • Zinkstok R.T.
      • Qiao Y.
      • Shahzad R.
      • Sokooti H.
      • et al.
      Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer.
      ProstateBladder, rectum, prostate, vs., nodes32≈220MonoCT/CTIn houseDLDSC, HD, MSD, DOSEMSD < 2–3 mmOne observerYesNo
      Anaya2019
      • Marin Anaya V.
      • Fairfoul J.
      Assessing the feasibility of adaptive planning for prostate radiotherapy using Smartadapt deformable image registration.
      ProstateProstate, rectum18126MultiCT/CBCTCommercialOtherVOL, DSC, CMS, MDC, DOSEDSC > 0.7 MDC < 2 mmOne observerYesNo
      An evocation is made, with references.
      Qiao2019

      Qiao Y, Jagt T, Hoogeman M, Lelieveldt BPF, Staring M. Evaluation of an Open Source Registration Package for Automatic Contour Propagation in Online Adaptive Intensity-Modulated Proton Therapy of Prostate Cancer. Front Oncol 2019;9. 10.3389/fonc.2019.01297.

      ProstateProstate, nodes, SV, rectum, bladder18159MonoCT/CTOpen sourceBSPDSC, MSD, 95%HD, DOSEMSD < 2 mmOne observer reviewed by a second observerYesNo
      Shaaer2019
      • Shaaer A.
      • Davidson M.
      • Semple M.
      • Nicolae A.
      • Mendez L.C.
      • Chung H.
      • et al.
      Clinical evaluation of an MRI-to-ultrasound deformable image registration algorithm for prostate brachytherapy.
      ProstateDIL1010MultiMR/USOpen sourceOther, BSPDSC, MDA, DBC, VOLNoFive observers and STAPLENoNo
      Eppenhof2020
      • Eppenhof K.a.J.
      • Maspero M.
      • Savenije MHF
      • de Boer JCJ
      • van der Voort van Zyp JRN
      • Raaymakers B.W.
      • et al.
      Fast contour propagation for MR-guided prostate radiotherapy using convolutional neural networks.
      ProstateProstate5100MonoMR/MRIn-house, open sourceDL, BSP95%HD, DSC, JACO, CMNoOne observerNoNo
      Hammers2020
      • Hammers J.E.
      • Pirozzi S.
      • Lindsay D.
      • Kaidar-Person O.
      • Tan X.
      • Chen R.C.
      • et al.
      Evaluation of a commercial DIR platform for contour propagation in prostate cancer patients treated with IMRT/VMAT.
      ProstateBladder, rectum20453Mono and multiCT/CT, CT/CBCTCommercial
      Research version.
      IntensityHD, MDA, DSC, JACNoOne observerNoNo
      Takagi2020
      • Takagi H.
      • Kadoya N.
      • Kajikawa T.
      • Tanaka S.
      • Takayama Y.
      • Chiba T.
      • et al.
      Multi-atlas-based auto-segmentation for prostatic urethra using novel prediction of deformable image registration accuracy.
      ProstatePU12020MonoCT/CTIn-houseAtlasCLD, DSCNoOne observerNoNo
      Liang2021
      • Liang X.
      • Bibault J.-E.
      • Leroy T.
      • Escande A.
      • Zhao W.
      • Chen Y.
      • et al.
      Automated contour propagation of the prostate from pCT to CBCT images via deep unsupervised learning.
      ProstateProstate5959MultiCT/CBCTIn-houseDLDSC, HD, MCC, COMNoFour observers on nine patients and STAPLENoNo
      Fu2021
      • Fu Y.
      • Wang T.
      • Lei Y.
      • Patel P.
      • Jani A.B.
      • Curran W.J.
      • et al.
      Deformable MR-CBCT prostate registration using biomechanically constrained deep learning networks.
      ProstateProstate, landmarks50100MultiMR/CBCTIn-houseDL, intensityDSC, MSD, TRENoAuto
      Automated generated contours.
      NoNo
      Ishida2021
      • Ishida T.
      • Kadoya N.
      • Tanabe S.
      • Ohashi H.
      • Nemoto H.
      • Dobashi S.
      • et al.
      Evaluation of performance of pelvic CT-MR deformable image registration using two software programs.
      ProstateProstate, bladder, rectum, FH510MultiCT/MROpen source, commercialFFD, BSPDSC, MDADSC > 0.8 MDA < 2 mmFive observersNoNo
      Masi2021
      • Masi M.
      • Landoni V.
      • Faiella A.
      • Farneti A.
      • Marzi S.
      • Guerrisi M.
      • et al.
      Comparison of rigid and deformable coregistration between mpMRI and CT images in radiotherapy of prostate bed cancer recurrence.
      ProstateRecurrence5353MultiCT/MRCommercialFFDDSC, JAC, DOSEDSC > 0.8 JACO > 0One observerYesNo
      McGeachy2021
      • McGeachy P.
      • Watt E.
      • Husain S.
      • Martell K.
      • Martinez P.
      • Sawhney S.
      • et al.
      MRI-TRUS registration methodology for TRUS-guided HDR prostate brachytherapy.
      ProstateProstate2020MultiMR/USCommercialOtherMDA, DSCNoOne observerNoNo
      Vozzo2021
      • Vozzo M.
      • Poder J.
      • Yuen J.
      • Bucci J.
      • Haworth A.
      Use of deformable image registration techniques to estimate dose to organs at risk following prostate external beam radiation therapy and high-dose-rate brachytherapy.
      ProstateProstate, rectum1010MultiCT/USCommercialMIMHHD, MDA, DSC, JACO, DOSEDSC > 0.8 MDA < 2 mmOne observerYesNo
      Kainz2022
      • Kainz K.
      • Garcia Alvarez J.
      • Zhong H.
      • Lim S.
      • Ahunbay E.
      • Tai A.
      • et al.
      Use of a DVH overlay technique for quality assurance of deformable image registration-based dose accumulation.
      ProstateProstate, bladder, rectum1010MultiCT/MVCTCommercialFFD, intensityDOSE, ICE, JACONoOne observerYesNo
      Faggiano2011
      • Faggiano E.
      • Fiorino C.
      • Scalco E.
      • Broggi S.
      • Cattaneo M.
      • Maggiulli E.
      • et al.
      An automatic contour propagation method to follow parotid gland deformation during head-and-neck cancer tomotherapy.
      H&NPG1010MultiMVCT/CTOpen sourceBSPASD, MSD, DSC, COM, VOLNoThree observersNoNo
      Hardcastle2012
      • Hardcastle N.
      • Tomé W.A.
      • Cannon D.M.
      • Brouwer C.L.
      • Wittendorp P.W.
      • Dogan N.
      • et al.
      A multi-institution evaluation of deformable image registration algorithms for automatic organ delineation in adaptive head and neck radiotherapy.
      H&NGTV, BS, PG, SC2222MonoCT/CTOpen source, in-houseDEM, SFBRDSC, MSHD, COM, QRGrading for QROne observerNoDSC/QR, MSHD/QR
      Mencarelli2012
      • Mencarelli A.
      • van Beek S.
      • van Kranen S.
      • Rasch C.
      • van Herk M.
      • Sonke J.-J.
      Validation of deformable registration in head and neck cancer using analysis of variance.
      H&N23 landmarks1078MultiCT/CBCTIn-houseIntensityDTANoTwo observersNoNo
      Leibfarth2013
      • Leibfarth S.
      • Mönnich D.
      • Welz S.
      • Siegel C.
      • Schwenzer N.
      • Schmidt H.
      • et al.
      A strategy for multimodal deformable image registration to integrate PET/MR into radiotherapy treatment planning.
      H&NSkin, bones, RT88MultiCT/MROpen sourceBSPJACO, DSC, NOF, MVD, NCCNoNANoNo
      Paganelli2013
      • Paganelli C.
      • Peroni M.
      • Riboldi M.
      • Sharp G.C.
      • Ciardo D.
      • Alterio D.
      • et al.
      Scale invariant feature transform in adaptive radiation therapy: a tool for deformable image registration assessment and re-planning indication.
      H&NMandible, PG, GTV1942Mono and multiCT/CT, CT/CBCTOpen sourceIntensityDSC, COM, SIFTNoOne observerNoNo
      Eiland2014
      • Eiland R.B.
      • Maare C.
      • Sjöström D.
      • Samsøe E.
      • Behrens C.F.
      Dosimetric and geometric evaluation of the use of deformable image registration in adaptive intensity-modulated radiotherapy for head-and-neck cancer.
      H&NGTV, nodes, PG, SC77MultiCT/CBCTCommercialDEMDSC, VOL, CMS, DOSENoOne observerYesNo
      Hoffmann2014
      • Hoffmann C.
      • Krause S.
      • Stoiber E.M.
      • Mohr A.
      • Rieken S.
      • Schramm O.
      • et al.
      Accuracy quantification of a deformable image registration tool applied in a clinical setting.
      H&N, Thorax, Abdomen30–50 landmarks1616MonoCT/CTCommercialBSPTRENoFive observersNoNo
      Huger2014
      • Huger S.
      • Graff P.
      • Harter V.
      • Marchesi V.
      • Royer P.
      • Diaz J.C.
      • et al.
      Evaluation of the Block Matching deformable registration algorithm in the field of head-and-neck adaptive radiotherapy.
      H&NMC, TG, HB, SMG1076MultiCT/CBCTCommercialOtherDSC, 95%HD, VOLNoTwo observers for three patientsNoNo
      Kumarasi2014
      • Kumarasiri A.
      • Siddiqui F.
      • Liu C.
      • Yechieli R.
      • Shah M.
      • Pradhan D.
      • et al.
      Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
      H&NTL, BS, SC, PG, eyes, chiasm, ON, cochlea, pituitary1010MonoCT/CTCommercial, open sourceBSP, DEMDSC, COM, HD, QRGrading for QROne observerNoDSC/COM, DSC/HD, COM/HD
      Mencarelli2014
      • Mencarelli A.
      • van Kranen S.R.
      • Hamming-Vrieze O.
      • van Beek S.
      • Nico Rasch C.R.
      • van Herk M.
      • et al.
      Deformable image registration for adaptive radiation therapy of head and neck cancer: accuracy and precision in the presence of tumor changes.
      H&N21 landmarks27102MultiCT/CBCTIn-houseBSPDTANoTwo observersNoNo
      Mohamed2014
      • Mohamed A.S.R.
      • Ruangskul M.-N.
      • Awan M.J.
      • Baron C.A.
      • Kalpathy-Cramer J.
      • Castillo R.
      • et al.
      Quality Assurance Assessment of Diagnostic and Radiation Therapy-Simulation CT Image Registration for Head and Neck Radiation Therapy: Anatomic Region of Interest–based Comparison of Rigid and Deformable Algorithms.
      H&N68 anatomic ROIs2020MonoCT/CTCommercial, open sourceAtlas, BSP, DEM, OFDSC, VO, 95%HD, FND, FPDNoThree observers for two patientsNoNo
      Garcia-Molla2015
      • García-Mollá R.
      • de Marco-Blancas N.
      • Bonaque J.
      • Vidueira L.
      • López-Tarjuelo J.
      • Perez-Calatayud J.
      Validation of a deformable image registration produced by a commercial treatment planning system in head and neck.
      H&N10 landmarks, GTV, BS, brain, larynx, jaw, SC, ON, eyes, PG515MultiCT/CBCTCommercialHybridTRE, ICE, DOSENoFour observersYesNo
      Ramadaan2015
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      H&NGTV, SC, PG, BS, VTB810MonoCT/CTCommercialDEMDSC, QRGrading for QRTwo observersNoNo
      Rigaud2015
      • Rigaud B.
      • Simon A.
      • Castelli J.
      • Gobeli M.
      • Ospina Arango J.-D.
      • Cazoulat G.
      • et al.
      Evaluation of Deformable Image Registration Methods for Dose Monitoring in Head and Neck Radiotherapy.
      H&N14 landmarks + PG, mandible, SC, TG, larynx, BS, brain1587MonoCT/CTOpen sourceDEM, FFDDSC, TRENoTwo observers for landmarksYesNo
      Hvid2016
      • Hvid C.A.
      • Elstrøm U.V.
      • Jensen K.
      • Alber M.
      • Grau C.
      Accuracy of software-assisted contour propagation from planning CT to cone beam CT in head and neck radiotherapy.
      H&NTongue, PG, PCM, SC, SMG, larynx, pharynx, esophagus3090Mono and multiCT/CBCT, CBCT/CBCTCommercialFFDDSC, HD, DOSEDSC > 0.80

      HD ≤ 3 mm

      0.8 > DSC > 0.7
      One observerYesNo
      Li2016
      • Li X.
      • Zhang Y.Y.
      • Shi Y.H.
      • Zhou L.H.
      • Zhen X.
      Evaluation of deformable image registration for contour propagation between CT and cone-beam CT images in adaptive head and neck radiotherapy.
      H&NPG, SMG631MultiCBCT/CTOpen sourceFFD, OF, DEM, LSDSC, PE, HDNoOne observerNoNo
      Zukauskaite2016
      • Zukauskaite R.
      • Brink C.
      • Hansen C.R.
      • Bertelsen A.
      • Johansen J.
      • Grau C.
      • et al.
      Open source deformable image registration system for treatment planning and recurrence CT scans.
      H&NSC, mandible, PG, SMG, TG, VTB2020MonoCT/CTOpen sourceDEMMSD, DSCNoOne observer contoured twiceNoNo
      Broggi2017
      • Broggi S.
      • Scalco E.
      • Belli M.L.
      • Logghe G.
      • Verellen D.
      • Moriconi S.
      • et al.
      A Comparative Evaluation of 3 Different Free-Form Deformable Image Registration and Contour Propagation Methods for Head and Neck MRI: The Case of Parotid Changes During Radiotherapy.
      H&NPG1212MonoMR/MRCommercial, open sourceFFD, BSPASD, MSD, DSC, ROCNoOne observerNoNo
      Li2017
      • Li X.
      • Zhang Y.
      • Shi Y.
      • Wu S.
      • Xiao Y.
      • Gu X.
      • et al.
      Comprehensive evaluation of ten deformable image registration algorithms for contour propagation between CT and cone-beam CT images in adaptive head & neck radiotherapy.
      H&NVTB, PG, SMG21129MultiCT/CBCTOpen source, in-houseOF, DEM, LS, BSPDSC, HDNoDouble check by one observerNoNo
      Yeap2017
      • Yeap P.L.
      • Noble D.J.
      • Harrison K.
      • Bates A.M.
      • Burnet N.G.
      • Jena R.
      • et al.
      Automatic contour propagation using deformable image registration to determine delivered dose to spinal cord in head-and-neck cancer radiotherapy.
      H&NSC3330 to 90MultiCT/MVCTOpen sourceBSPDBC, CI, DTC, DOSENoSix observers on several patientsYesNo
      Nix2017
      • Nix M.G.
      • Prestwich R.J.D.
      • Speight R.
      Automated, reference-free local error assessment of multimodal deformable image registration for radiotherapy in the head and neck.
      H&NGTV, BS1428MultiCT/MRIn-houseIntensityEVF, ROCNoOne observerNoNo
      Nobnop2017
      • Nobnop W.
      • Chitapanarux I.
      • Neamin H.
      • Wanwilairat S.
      • Lorvidhaya V.
      • Sanghangthum T.
      Evaluation of Deformable Image Registration (DIR) Methods for Dose Accumulation in Nasopharyngeal Cancer Patients during Radiotherapy.
      H&NGTV, PG, SC530MonoMVCT/MVCTOpen sourceDIRARTDSC, ICE, DOSEDSC > 0,7One observerYesNo
      Nobnop2017
      • Nobnop W.
      • Neamin H.
      • Chitapanarux I.
      • Wanwilairat S.
      • Lorvidhaya V.
      • Sanghangthum T.
      Accuracy of eight deformable image registration (DIR) methods for tomotherapy megavoltage computed tomography (MVCT) images.
      H&NPG77MonoMVCT/MVCTOpen sourceDIRARTCC, DSCNoOne observerNoNo
      Taylor2018
      • Taylor A.
      • Sen M.
      • Prestwich R.J.D.
      Assessment of the Impact of Deformable Registration of Diagnostic MRI to Planning CT on GTV Delineation for Radiotherapy for Oropharyngeal Carcinoma in Routine Clinical Practice.
      H&NGTV2222MultiCT/MRCommercialIntensityCI,MDC, DSC, CGDNoOne observerNoNo
      Zhang2018
      • Zhang L.
      • Wang Z.
      • Shi C.
      • Long T.
      • Xu X.G.
      The impact of robustness of deformable image registration on contour propagation and dose accumulation for head and neck adaptive radiotherapy.
      H&NGTV, PG, SC, BS, TL, lens, ON, cochlea10NAMonoCT/CTCommercialAnaconda, MorfeusDSC, HD, VOL, DOSENoOne observerYesNo
      Nobnop2019

      Nobnop W, Chitapanarux I, Wanwilairat S, Tharavichitkul E, Lorvidhaya V, Sripan P. Effect of Deformation Methods on the Accuracy of Deformable Image Registration From Kilovoltage CT to Tomotherapy Megavoltage CT. Technol Cancer Res Treat 2019;18:1533033818821186. 10.1177/1533033818821186.

      H&NGTV, PG, SC1212Mono & multiCT/CT, MVCT/MVCT, CT/MVCTOpen sourceDIRARTDSC, nMI, ICEDSC > 0.7

      nMI > 1
      Three observersNoNo
      Mee2020
      • Mee M.
      • Stewart K.
      • Lathouras M.
      • Truong H.
      • Hargrave C.
      Evaluation of a deformable image registration quality assurance tool for head and neck cancer patients.
      H&NPG, mandible3535MonoCT/CTCommercialOtherHD, MDA, DSC, JACO, QRHD < 3 mm MDA < 2–3 mm DSC > 0.8–0.9

      JACO > 0
      Five observers on several patients for QRNoNo
      Christiansen2021
      • Christiansen R.L.
      • Johansen J.
      • Zukauskaite R.
      • Hansen C.R.
      • Bertelsen A.S.
      • Hansen O.
      • et al.
      Accuracy of automatic structure propagation for daily magnetic resonance image-guided head and neck radiotherapy.
      H&NGTV, nodes, BS, SC, PG, SMG, TG1717MultiCT/MRCommercialAtlasDSC, MSD, HDNoOne observer contoured twiceNoNo
      Yang2022
      • Yang B.
      • Chen X.
      • Li J.
      • Zhu J.
      • Men K.
      • Dai J.
      A feasible method to evaluate deformable image registration with deep learning–based segmentation.
      H&NCTV, BS, SC, PG, TG, larynx, trachea, mandible2020MonoCT/CTCommercialFFDDSC, MDA, TREDSC > 0.8 MDA < 2 mmAuto
      Automated generated contours.
      NoNo
      Nash2022
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      H&NSC, BS, PG, larynx1050MultiCT/CBCTCommercialDEM, FFD, Anaconda, AdmireDCS, HD, DTA, CM, DOSEDSC > 0.8One observer and STAPLE. One observer contoured twice 3 patientsYesNo
      Gaede2011
      • Gaede S.
      • Olsthoorn J.
      • Louie A.V.
      • Palma D.
      • Yu E.
      • Yaremko B.
      • et al.
      An evaluation of an automated 4D-CT contour propagation tool to define an internal gross tumour volume for lung cancer radiotherapy.
      ThoraxGTV, nodes10100MonoCT/CTIn-houseOtherCM, DSCDSC > 0.7Six observersNoNo
      Speight2011
      • Speight R.
      • Sykes J.
      • Lindsay R.
      • Franks K.
      • Thwaites D.
      The evaluation of a deformable image registration segmentation technique for semi-automating internal target volume (ITV) production from 4DCT images of lung stereotactic body radiotherapy (SBRT) patients.
      ThoraxGTV25300MonoCT/CTCommercialBSP, DEMDSC, MDA, nDSCnDSC < 1 mmOne observer contoured twiceNoNo
      Peroni2013
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      ThoraxLung, SC, heart, esophagus33MonoCT/CTOpen sourceBSPDSC, VOL, PPV, SDNoFive observers and STAPLENoDSC/VOL, DSC/SD, DSC/PPV, SD/VOL, SD/PPV, VOL/PPV
      Balik2013
      • Balik S.
      • Weiss E.
      • Jan N.
      • Roman N.
      • Sleeman W.C.
      • Fatyga M.
      • et al.
      Evaluation of 4-dimensional Computed Tomography to 4-dimensional Cone-Beam Computed Tomography Deformable Image Registration for Lung Cancer Adaptive Radiation Therapy.
      ThoraxGTV8204MultiCT/CBCTOpen sourceDEM, otherDSC, ASD, FP, FNNoTwo observers for one patientNoNo
      Hardcastle2013
      • Hardcastle N.
      • van Elmpt W.
      • De Ruysscher D.
      • Bzdusek K.
      • Tomé W.A.
      Accuracy of deformable image registration for contour propagation in adaptive lung radiotherapy.
      ThoraxGTV, nodes, lungs, esophagus, SC1717MonoCT/CTCommercial, in-house, open sourceDEM, SFBR, otherDSC, MSHD, COM, QRGrading for QROne observerNoDSC/QR, MSHD/QR
      Ottosson2014
      • Ottosson W.
      • Andersen J.A.L.
      • Borrisova S.
      • Mellemgaard A.
      • Behrens C.F.
      Deformable image registration for geometrical evaluation of DIBH radiotherapy treatment of lung cancer patients.
      ThoraxGTV, body, SC, lungs, esophagus, heart3312Mono and multiCT/CT, CT/CBCTCommercialDEMDSC, VOLNoOne observerNoNo
      Hardcastle2015
      • Hardcastle N.
      • Hofman M.S.
      • Hicks R.J.
      • Callahan J.
      • Kron T.
      • MacManus M.P.
      • et al.
      Accuracy and Utility of Deformable Image Registration in 68Ga 4D PET/CT Assessment of Pulmonary Perfusion Changes During and After Lung Radiation Therapy.
      Thorax8 landmarks + lungs515MonoCT/CTOpen sourceBSP, DEMMSHD, DSC, TRENoOne observerYesNo
      Stützer2016
      • Stützer K.
      • Haase R.
      • Lohaus F.
      • Barczyk S.
      • Exner F.
      • Löck S.
      • et al.
      Evaluation of a deformable registration algorithm for subsequent lung computed tomography imaging during radiochemotherapy.
      Thorax25 landmarks + lung1554MonoCTIn-houseOtherJAC, MDA, MCD, HD, GCD, TRE, IVF, VOLNoOne observerNoMCD/JAC, GCD/JAC, MCD/GCD
      Guo2017
      • Guo Y.
      • Li J.
      • Zhang P.
      • Shao Q.
      • Xu M.
      • Li Y.
      Comparative evaluation of target volumes defined by deformable and rigid registration of diagnostic PET/CT to planning CT in primary esophageal cancer.
      ThoraxGTVs1919MonoCT/CTCommercialFFDDSC, COMNoOne observerNoNo
      Jin2017
      • Jin R.
      • Liu Y.
      • Chen M.
      • Zhang S.
      • Song E.
      Contour propagation for lung tumor delineation in 4D-CT using tensor-product surface of uniform and non-uniform closed cubic B-splines.
      ThoraxGTV1122MonoCT/CTIn-houseBSPJSC, HDNoThree observersNoNo
      Jin2017
      • Jin R.
      • Liu Y.
      • Chen M.
      • Zhang S.
      • Song E.
      Contour propagation for lung tumor delineation in 4D-CT using tensor-product surface of uniform and non-uniform closed cubic B-splines.
      ThoraxGTV1818MonoCT/CTIn-houseBSPJACO, HDNoOne observerNoNo
      Ma2017
      • Ma Y.
      • Liu X.
      • Dai Z.
      • He P.
      • Yan Y.
      • Shen G.
      • et al.
      Evaluation of mesh- and binary-based contour propagation methods in 4D thoracic radiotherapy treatments using patient 4D CT images.
      ThoraxGTV, lungs, hearts, SC66MonoCT/CTIn-houseDEMDSC, COM, MSHDNoOne observerNoNo
      Moriya2017
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ThoraxGTV1919MonoCT/CTCommercial, in-houseFFD, BSP, IntensityDOSE, DSCDSC > 0,7One observer double checked by a second observerYesDSC/DOSE
      Sugawara2017
      • Sugawara Y.
      • Tachibana H.
      • Kadoya N.
      • Kitamura N.
      • Sawant A.
      • Jingu K.
      Prognostic factors associated with the accuracy of deformable image registration in lung cancer patients treated with stereotactic body radiotherapy.
      ThoraxGTV, lung, body1919MonoCT/CTOpen source, commercialFFD, BSP, IntensityDSC, nDSC1DSC > 0.7

      nDSC > 1 mm
      One observerNoNo
      Yan2017
      • Yan P.
      • Kodera Y.
      • Shimamoto K.
      Evaluation of Deformable Image Registration for Three-Dimensional Temporal Subtraction of Chest Computed Tomography Images.
      Thorax300–600 landmarks + lungs1010MonoCT/CTOpen sourceIntensityJACO, TRENoOne observerNoNo
      Zhang2018
      • Zhang J.
      • Markova S.
      • Garcia A.
      • Huang K.
      • Nie X.
      • Choi W.
      • et al.
      Evaluation of automatic contour propagation in T2-weighted 4DMRI for normal-tissue motion assessment using internal organ-at-risk volume (IRV).
      ThoraxLungs, heart, liver, stomach1090MonoMR/MRIn-houseFFDJAC, SENS, SPE, VOLNoTwo observers and STAPLENoNo
      Azcona2019
      • Azcona J.D.
      • Huesa-Berral C.
      • Moreno-Jiménez M.
      • Barbés B.
      • Aristu J.J.
      • Burguete J.
      A novel concept to include uncertainties in the evaluation of stereotactic body radiation therapy after 4D dose accumulation using deformable image registration.
      Thorax65 landmarks, GTV1111MonoCT/CTCommercialFFDTRE, DOSENoOne observerYesNo
      Sarudis2019
      • Sarudis S.
      • Karlsson A.
      • Bibac D.
      • Nyman J.
      • Bäck A.
      Evaluation of deformable image registration accuracy for CT images of the thorax region.
      ThoraxGTV, heart, lungs, SC660MonoCT/CTCommercialAnaconda, BSP, DEMDSC, CMS, DOSEDSC > 0.8One observerYesDSC/DOSE
      Lei2020
      • Lei Y.
      • Fu Y.
      • Wang T.
      • Liu Y.
      • Patel P.
      • Curran W.J.
      • et al.
      4D-CT deformable image registration using multiscale unsupervised deep learning.
      ThoraxLandmarks2545MonoCT/CTIn-house, commercialDL, BSPTRENoOne observerNoNo
      He2021
      • He Y.
      • Cazoulat G.
      • Wu C.
      • Peterson C.
      • McCulloch M.
      • Anderson B.
      • et al.
      Geometric and dosimetric accuracy of deformable image registration between average-intensity images for 4DCT-based adaptive radiotherapy for non-small cell lung cancer.
      ThoraxGTV, lungs, landmarks28112MonoCT/CTCommercialAnacondaDSC, TRE, DOSETRE < voxel sizeOne observerYesNo
      Nenoff2021
      • Nenoff L.
      • Matter M.
      • Amaya E.J.
      • Josipovic M.
      • Knopf A.-C.
      • Lomax A.J.
      • et al.
      Dosimetric influence of deformable image registration uncertainties on propagated structures for online daily adaptive proton therapy of lung cancer patients.
      ThoraxCTV545MonoCT/CTOpen source, commercialBSP, DEM, Anaconda, morfeus, FFDVOL, DOSENoOne observerYesNo
      Eijnatten2021
      • van Eijnatten M.
      • Rundo L.
      • Batenburg K.J.
      • Lucka F.
      • Beddowes E.
      • Caldas C.
      • et al.
      3D deformable registration of longitudinal abdominopelvic CT images using unsupervised deep learning.
      Thorax, abdomen, cervixVTB12?MonoCT/CTIn-house, open-sourceDL, FFDDSC, SSIMNoOne observerNoNo
      Han2022

      Han MC, Kim J, Hong C-S, Chang KH, Han SC, Park K, et al. Performance Evaluation of Deformable Image Registration Algorithms Using Computed Tomography of Multiple Lung Metastases. Technol Cancer Res Treat 2022;21:15330338221078464. 10.1177/15330338221078464.

      ThoraxGTV, landmarks550MonoCT/CTOpen-sourceDIRARTTRENoOne observer reviewed by a second observerNoNo
      Omidi2022
      • Omidi A.
      • Weiss E.
      • Wilson J.S.
      • Rosu-Bubulac M.
      Quantitative assessment of intra- and inter-modality deformable image registration of the heart, left ventricle, and thoracic aorta on longitudinal 4D-CT and MR images.
      ThoraxHeart, LV, TA588Mono and multiCT/CT, MR/MR, CT/MRCommercialFFDDSC, MDA, HDDSC > 0.8 MDA < 3 mmOne observer reviewed by a second observerNoNo
      Jung2013
      • Hoon Jung S.
      • Min Yoon S.
      • Ho Park S.
      • Cho B.
      • Won Park J.
      • Jung J.
      • et al.
      Four-dimensional dose evaluation using deformable image registration in radiotherapy for liver cancer.
      Abdomen21 landmarks + GTV, liver, duodenum, stomach, kidneys1111MonoCT/CTOpen sourceDIRARTTRE, DSC, DOSENoOne observerYesNo
      Fukumitsu2017
      • Fukumitsu N.
      • Nitta K.
      • Terunuma T.
      • Okumura T.
      • Numajiri H.
      • Oshiro Y.
      • et al.
      Registration error of the liver CT using deformable image registration of MIM Maestro and Velocity AI.
      AbdomenLandmarks2424MonoCT/CTCommercialBSP, intensityTRENoOne observerNoNo
      Lee2018
      • Lee S.
      • Kim H.
      • Ji Y.
      • Cho B.
      • Kim S.S.
      • Jung J.
      • et al.
      Evaluation of Hepatic Toxicity after Repeated Stereotactic Body Radiation Therapy for Recurrent Hepatocellular Carcinoma using Deformable Image Registration.
      Abdomenliver8585MonoCT/CTCommercialFFDDSC, DOSENoOne observerYesNo
      Kubota2019
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      AbdomenCTV, stomach, duodenum, bowel618MonoCT/CTCommercialIntensity, MIMHDSC, MDA, DOSEMDA < 2 mmOne observerYesMDA/DSC, MDA/DOSE
      Sen2020
      • Sen A.
      • Anderson B.M.
      • Cazoulat G.
      • McCulloch M.M.
      • Elganainy D.
      • McDonald B.A.
      • et al.
      Accuracy of deformable image registration techniques for alignment of longitudinal cholangiocarcinoma CT images.
      Abdomen5 landmarks2929MonoCT/CTCommercialDEM, BSP, SFBR, anaconda, morfeusTRENoOne observerNoNo
      Zhang2020
      • Zhang Y.
      • Paulson E.
      • Lim S.
      • Hall W.A.
      • Ahunbay E.
      • Mickevicius N.J.
      • et al.
      A Patient-Specific Autosegmentation Strategy Using Multi-Input Deformable Image Registration for Magnetic Resonance Imaging-Guided Online Adaptive Radiation Therapy: A Feasibility Study.
      AbdomenLiver, kidneys, spleen, aorta, pancreas, stomach, duodenum, bowel, colon1060MonoMR/MRCommercial
      Research version.
      AdmireDSC, MDADSC > 0.8 MDA < 2 mmThree observersNoNo
      Anderson2021
      • Anderson B.M.
      • Lin Y.-M.
      • Lin E.Y.
      • Cazoulat G.
      • Gupta S.
      • Kyle Jones A.
      • et al.
      A novel use of biomechanical model-based deformable image registration (DIR) for assessing colorectal liver metastases ablation outcomes.
      AbdomenGTV, ablation zone3030MonoCT/CTCommercialMorfeusDTA, VOLNoOne observer reviewed by a second observerNoNo
      Zambra2013
      • Zambrano V.
      • Furtado H.
      • Fabri D.
      • LÜtgendorf-Caucig C.
      • GÓra J.
      • Stock M.
      • et al.
      Performance validation of deformable image registration in the pelvic region.
      CervixRectum, bladder1020Mono and multiCT/CT, CT/CBCTIn-houseIntensityDSCNoOne observerNoNo
      Abe2014
      • Abe T.
      • Tamaki T.
      • Makino S.
      • Ebara T.
      • Hirai R.
      • Miyaura K.
      • et al.
      Assessing cumulative dose distributions in combined radiotherapy for cervical cancer using deformable image registration with pre-imaging preparations.
      CervixRectum, bladder, CTV520MonoCT/CTCommercialDEMDSC, DOSENoOne observerYesNo
      Kadoya2017
      • Kadoya N.
      • Miyasaka Y.
      • Yamamoto T.
      • Kuroda Y.
      • Ito K.
      • Chiba M.
      • et al.
      Evaluation of rectum and bladder dose accumulation from external beam radiotherapy and brachytherapy for cervical cancer using two different deformable image registration techniques.
      CervixUterus, rectum, bladder1144MonoCT/CTCommercialHybrid, intensityDSC, DOSENoOne observerYesNo
      Chapmaan2018
      • Chapman C.H.
      • Polan D.
      • Vineberg K.
      • Jolly S.
      • Maturen K.E.
      • Brock K.K.
      • et al.
      Deformable image registration–based contour propagation yields clinically acceptable plans for MRI-based cervical cancer brachytherapy planning.
      CervixBladder, CTV, rectum, sigmoid1042MonoIRM/IRMCommercialBSPDSC, MDA, DOSEDSC > 0.8One observer contoured twiceYesNo
      Dyer2019
      • Dyer B.A.
      • Yuan Z.
      • Qiu J.
      • Benedict S.H.
      • Valicenti R.K.
      • Mayadev J.S.
      • et al.
      Factors associated with deformation accuracy and modes of failure for MRI-optimized cervical brachytherapy using deformable image registration.
      CervixCTV, cervix, uterus2626MultiCT/MRCommercialAnaconda, MorfeusDSC, DTA, COM,

      angle, VOL, DVFmax, ROC, QR
      NoOne observer reviewed by a second observerNoDSC/QR, DTA/QR
      Mohamadi2019
      • Mohammadi R.
      • Mahdavi S.R.
      • Jaberi R.
      • Siavashpour Z.
      • Janani L.
      • Meigooni A.S.
      • et al.
      Evaluation of deformable image registration algorithm for determination of accumulated dose for brachytherapy of cervical cancer patients.
      CervixBladder, rectum137274MonoCT/CTCommercialMIMHDSC, JACO, HD, MDANoOne observerYesNo
      White2019
      • White I.
      • McQuaid D.
      • McNair H.
      • Dunlop A.
      • Court S.
      • Hopkins N.
      • et al.
      Geometric and dosimetric evaluation of the differences between rigid and deformable registration to assess interfraction motion during pelvic radiotherapy.
      CervixCTV, rectum, bladder17169MultiCT/CBCTOpen sourceAnacondaDSC, MDAMDA < 2–3 mm DSC > 0.8–0.9One observerNo
      Not directly for DIR performance quantification.
      No
      Zeng2020

      Zeng J, Chen J, Zhang D, Meng M, Zhang B, Qu P, et al. Assessing cumulative dose distributions in combined external beam radiotherapy and intracavitary brachytherapy for cervical cancer by treatment planning based on deformable image registration. Translational Cancer Research 2020;9. 10.21037/tcr-20-1196.

      CervixCTVs, rectum, bladder2323MonoCT/CTCommercialOtherDOSENoOne observerYesNo
      Lee2019
      • Lee S.
      • Stewart J.
      • Lee Y.
      • Myrehaug S.
      • Sahgal A.
      • Ruschin M.
      • et al.
      Improved dosimetric accuracy with semi-automatic contour propagation of organs-at-risk in glioblastoma patients undergoing chemoradiation.
      BrainBS, ON, eyes, chiasm1442MonoMR/MRCommercial
      Research version.
      AdmireDSC, HD, DOSENoOne observerYesNo
      Akbarzadeh2013
      • Akbarzadeh A.
      • Gutierrez D.
      • Baskin A.
      • Ay M.R.
      • Ahmadian A.
      • Alam N.R.
      • et al.
      Evaluation of whole-body MR to CT deformable image registration.
      Whole body16 landmarks + brain, lungs, kidneys, body2828MultiCT/MROpen sourceBSPTO, DSC, JAC, HD, DEGrading for DSCOne observerNoNo
      95%HD: 95th percentile of the Hausdorff distance; ANOVA: analysis of variance method; ASD: average symmetric distance; ASuD: average surface distance; BS: brainstem; BSP: B-spline; CBCT: cone-beam CT; CC: correlation coefficient; CD: contour distance; CGD: center of gravity distance; CI: conformity index; CLD: centerlines distance; CM: centroid motion; CMS: center of mass shift; COM: center of mass; Con: conventional; CT: computed tomography; CTV: clinical target volume; DBC: distance between centroids; DDM: distance discordance metric; DE: distance error; DEM: demons; DIL: dominant intraprostatic lesion; DOSE: dose-based parameters; DSC: Dice similarity coefficient; DTA: distance to agreement; DTC: distance to conformity; DVFmax: maximum deformation vector field; EVF: error vector field; FEM: finite element model; FFD: free-form deformation; FH: femoral heads; FN: false negative; FND: false-negative Dice; FP: false-positive; FPD: false-positive Dice; GCD: geometric center distance; GTV: Gross Tumor Volume; HB: hyoid bone; HD: Hausdorff distance; H&N: head-and-neck; Hyb: Hybrid; ICE: inverse consistency error; IVF: inconsistency vector field; JAC: Jaccard coefficient; JACO: Jacobian determinant; LS: level-set; LV: left ventricle; MC: medullary canal; MCC: Matthews correlation coefficient; MCD: mean contour distance; MCS: mean contour separation; MDA: mean distance to agreement; MDC: mean distance to conformity; MI: mutual information; MIMH: commercial MIM hybrid algorithm; MR: magnetic resonance; MSD: maximum symmetric distance; MSE: root mean square error; MSHD: mean of the slicewise Hausdorff distances; MVCT: megavoltage computed tomography; MVD: mean volume distance; NCC: normalized cross correlation; nDCS: normalized DSC; nMI: normalized mutual information; NOF: nonoverlapping fraction; NTCP: normal tissue complication probability; OF: optical flow; ON: optic nerves; PB: penile bulb; PCM: pharyngeal constrictor muscle; PG: parotid glands; PPV: positive predictive value; PU: prostatic urethra; QR: qualitative rating; ROC: receiver operating characteristic; ROI: regions of interest; RT: respiratory tract; SC: spinal cord; SCR: symmetric correlation ratio; SD: surface distance; SENS: sensitivity; SFBR: salient-feature-based registration; SIFT: scale invariant feature transform; SMG: submandular glands; SPE: specificity; SSIM: structural similarity index; STAPLE: simultaneous truth and performance level estimation; SV: seminal vesicles; TA: thoracic aorta; TC: Tannimoto Coefficient; TE: transitivity error; TG: thyroid gland; TL: temporal lobe; TO: target overlap; TRE: target registration error; US: ultrasound; VO: volume overlap; VOL: volume analysis (for example volume comparison); VTB: vertebras.
      * Research version.
      ** Not directly for DIR performance quantification.
      *** An evocation is made, with references.
      **** Automated generated contours.

      Number of patients and registrations

      The mean number of patients per publication was 20 (standard deviation (SD) = 21), with a median of 12 and a maximum number of patients included of 137 (Fig. 2): the vast majority of the work was therefore based on the use of a limited number of patients, with only 10/107 publications exceeding 50 patients. The mean and median DIRs performed per publication were 63 and 29 (Fig. 2), respectively (min–max: 3–800; SD = 103).
      Figure thumbnail gr2
      Fig. 2Boxplot of the number of patients included (left) and registrations performed (right) for the 107 articles in our review.

      Localizations

      With 95 publications, prostate (36), head-and-neck (H&N) (33), and thoracic (26) locations represented more than ¾ of the studied locations (Fig. 3). Cervix (10), abdominal (12) and cerebral (2) locations were studied more marginally.
      Figure thumbnail gr3
      Fig. 3Number of articles by studied localization (H&N: head-and-neck).

      Structures and landmarks used as ground-truth

      Details of the anatomical structures delineated to quantify the quality of the DIR are given in Fig. 4. Forty-eight different organs at risk (OARs) were used: the most mentioned were the rectum, bladder, parotid glands, prostate, spinal canal and lungs in 26, 24, 23, 24, 20 and 13 publications, respectively. The deformation of target volumes (GTV or CTV) or lymph nodes was studied in 43 publications. Delineation of anatomical structures was used as ground truth to quantify DIR quality in 96 publications, with landmarks being used in 23 articles (Fig. 5).
      Figure thumbnail gr4
      Fig. 4Number of publications by type of delineated organ.
      Figure thumbnail gr5
      Fig. 5Number of publications by type of ground truth.

      Inter- and intraobserver variability

      Establishing the ground truth is not trivial because of the known problem of inter- and intraobserver variability in delineating anatomical structures. This variability has been considered in 39 articles (Fig. 6), and a single definition of the anatomical structure and/or landmark was used as ground truth in the other 68 articles.
      Figure thumbnail gr6
      Fig. 6Number of publications that characterized or did not characterize intra- or interobserver variability in the delineation.

      Imaging modalities

      Sixty-seven articles were devoted to monomodal registration and 49 to multimodal registration (Fig. 7). CT/CT and CT/CBCT registrations were the most studied in each category, with 55 and 23 articles, respectively.
      Figure thumbnail gr7
      Fig. 7Number of publications by registered imaging modalities (CT: computed tomography; CBCT: cone-beam CT; MVCT: megavoltage CT; MR: magnetic resonance; US: ultrasound).

      DIR methods

      The DIR methods used came from commercial, open-source or in-house solutions in 62, 34 and 27 publications, respectively. Intensity-based DIRs were evaluated 118 times (mainly B-spline, demons and free-form deformations), compared to 26 times for hybrid and six times for deep-learning-based methods (see Table 2 for details). These results can be explained by the more recent development of hybrid and DL-based methods. Note that it is possible to refer to two other reviews for articles devoted to DL-based DIR [
      • Fu Y.
      • Lei Y.
      • Wang T.
      • Curran W.J.
      • Liu T.
      • Yang X.
      Deep learning in medical image registration: a review.
      ,
      • Xiao H.
      • Ren G.
      • Cai J.
      A review on 3D deformable image registration and its application in dose warping.
      ], which we have often not retained because of the frequent use of public or synthetic database.
      Table 2Number of times a DIR method was evaluated in the articles of our review, classified by the DIR nature.
      Intensity-based DIRHybrid-basedDLAtlas
      BSDEMFFDOFLSDIRARTSFBRNSANAMORADMMIMHNS
      38301942531711634263
      ADM: admire; ANA: anaconda; BS: B-spline; DEM: demons; DL: deep-learning; FFD: free-form deformation; LS: level-set; MIMH: MIM hybrid module; MOR: morfeus; NP: not specified; OF: optical flow; SFBR: salient feature based.

      Metrics

      We initially identified 61 metrics with different names in the 107 articles. We deliberately chose to discard metrics that appeared only once. We have also merged different names that designate an identical metric; we kept MDA (mean distance to agreement) terminology to refer to MDA and MDC (mean distance to conformity) since they both measured the same quantity. Similarly, the metrics COM (center of mass displacement), CMS (center of mass shift) and DBC (distance between centers) are referred to as COM. We finally identified 23 different geometric metrics used to characterize the performance of a DIR and appeared at least twice (Fig. 8). Notably, the dosimetric impact of DIR was quantified in 32 articles.
      Figure thumbnail gr8
      Fig. 8Number of publications according to the metric used (only metrics appearing at least twice are represented). 95%HD: 95th percentile of the Hausdorff distance; ASD: average symmetric distance; CC: correlation coefficient; CI: conformity index; CM: centroid motion; COM: center of mass; DSC: Dice similarity coefficient; DTA: distance to agreement; HD: Hausdorff distance; ICE: inverse consistency error; JAC: Jaccard coefficient; JACO: Jacobian; MDA: mean distance to agreement; MI: mutual information; MSD: maximum symmetric distance; MSHD: mean of the slicewise Hausdorff distances; nDSC: normalized DSC; PPV: positive predictive value; QR: qualitative rating; ROC: receiver operating characteristic; SENS: sensitivity; TRE: target registration error; VOL: volume analysis (for example volume comparison).
      In order to get an idea of metrics practices, Fig. 9 illustrates the evolution per year of the most used metrics between 2010 and 2022. Similarly, Fig. 10 shows how the use of tolerance levels for metrics evolved between 2010 and 2022.
      Figure thumbnail gr9
      Fig. 9Percentage of papers published per year per metric. 95%HD: 95th percentile of the Hausdorff distance; COM: center of mass; DSC: Dice similarity coefficient; DOSE: dose-based parameters; HD: Hausdorff distance; JACO: Jacobian; MDA: mean distance to agreement; MSHD: mean of the slicewise Hausdorff distances; TRE: target registration error.
      Figure thumbnail gr10
      Fig. 10Percentage of articles by year of publication with or without threshold, or by qualitative rating.

      Discussion

      Geometric metrics

      The 23 identified metrics were comparable in number and type to the 20 metrics reported in a literature review dedicated to image segmentation [
      • Taha A.A.
      • Hanbury A.
      Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool.
      ]. Some differences can, however, be noted. Some metrics of Taha's review whose references are prior to 2010 do not appear in our article, while others that we listed, such as QR (qualitative ratings) or TRE (target registration error), are not the subject of Taha's review, which only targets geometric metrics applicable to 3D structures.
      The Dice similarity coefficient (DSC) is by far the most used metric in our review, as it is quantified in 82/107 articles (Fig. 8). Hausdorff distance (HD), COM, MDA, and TRE are used in >10 articles. Simple volumetric comparisons (VOL) were performed in 18 papers.
      Following the classification of Taha et al., these 23 metrics can be categorized into the following five classes: overlap based for conformity index (CI), DSC, nDSC (normalized DSC), JAC (Jaccard coefficient), PPV (positive predictive value) and SENS (sensitivity); volume-based for JACO (Jacobian determinant), VOL and QR; information theory based for MI (mutual information); probabilistic based for CC (correlation coefficient) and ROC (receiver operating characteristic); spatial distance-based for ASD (average symmetric distance), COM, CM (centroid motion), DTA (distance to agreement), 95%HD, HD, MDA, MSD (maximum symmetric distance), MSHD (mean of the slicewise Hausdorff distances), TRE and ICE (inverse consistency error). In the 107 articles, 54% of the authors combined metrics from the overlap-based and spatial distance-based categories, and 25% completed with an additional metric from the volume-based category. DSC, HD and COM were the most combined metrics, and VOL often completed the used metrics. Indeed, complementary measures such as COM, HD, and DSC are useful for better characterizing the quality of volume overlaps [
      • Kumarasiri A.
      • Siddiqui F.
      • Liu C.
      • Yechieli R.
      • Shah M.
      • Pradhan D.
      • et al.
      Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
      ].
      In the last ten years, strong evolution in metrics was not observed (Fig. 9). There has been a slight decrease in the use of DSC in favor of HD. These two types of metrics were often combined, and the volumetric overlap and distance metrics were often not highly correlated and, therefore, potentially complementary [
      • Sherer M.V.
      • Lin D.
      • Elguindi S.
      • Duke S.
      • Tan L.-T.
      • Cacicedo J.
      • et al.
      Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: a critical review.
      ]. There has also been an increase in MDA use since the publication of AAPM TG132 (see Chapter 4.2). Finally, regarding dosimetric metrics, their use seems to be increasing since 2013, which can be explained by the growing interest in DIR in dose accumulation applications.

      Dosimetric metrics

      Dosimetric metrics were used in 32 papers. Table 3 gives details of applications and dosimetric metrics used. These applications can be separated into two categories [
      • Rigaud B.
      • Simon A.
      • Castelli J.
      • Lafond C.
      • Acosta O.
      • Haigron P.
      • et al.
      Deformable image registration for radiation therapy: principle, methods, applications and evaluation.
      ]. The first one consists in propagating a contour from one examination to another on which the dose is recalculated, for example from a CT to a CBCT to estimate the dose fraction. The second application concerns dose accumulation, typically in ART, which consists in applying the DVF to the dose matrix to map the dose from one examination to another. In the majority of cases, the dosimetric impact of the DIR was quantified by comparing the difference between Dx/Vy (where Dx is the dose delivered in x% of the volume of a structure and Vy is the volume of the structure receiving a y dose). This type of comparison is performed in 19 and 27 articles for targets and OARs, respectively. The authors used various Dx/Vy, even for identical structures, making it difficult to compare the results. Indeed, in prostate treatment, the dosimetric metrics chosen for the rectum were V70 Gy [
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ], D2cc [
      • Vozzo M.
      • Poder J.
      • Yuen J.
      • Bucci J.
      • Haworth A.
      Use of deformable image registration techniques to estimate dose to organs at risk following prostate external beam radiation therapy and high-dose-rate brachytherapy.
      ] or V50 Gy and V60 Gy [
      • Marin Anaya V.
      • Fairfoul J.
      Assessing the feasibility of adaptive planning for prostate radiotherapy using Smartadapt deformable image registration.
      ]. Biological metrics were more rarely used: the general equivalent uniform dose (gEUD) in four articles [
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ,
      • Thor M.
      • Andersen E.S.
      • Petersen J.B.B.
      • Sørensen T.S.
      • Noe K.Ø.
      • Tanderup K.
      • et al.
      Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy.
      ,
      • Hoon Jung S.
      • Min Yoon S.
      • Ho Park S.
      • Cho B.
      • Won Park J.
      • Jung J.
      • et al.
      Four-dimensional dose evaluation using deformable image registration in radiotherapy for liver cancer.
      ,
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ] and the normal tissue complication probability (NTCP) in two [
      • Thor M.
      • Andersen E.S.
      • Petersen J.B.B.
      • Sørensen T.S.
      • Noe K.Ø.
      • Tanderup K.
      • et al.
      Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy.
      ,
      • Hoon Jung S.
      • Min Yoon S.
      • Ho Park S.
      • Cho B.
      • Won Park J.
      • Jung J.
      • et al.
      Four-dimensional dose evaluation using deformable image registration in radiotherapy for liver cancer.
      ]. Dosimetric indices such as homogeneity or conformity indices were used in five articles [
      • Marin Anaya V.
      • Fairfoul J.
      Assessing the feasibility of adaptive planning for prostate radiotherapy using Smartadapt deformable image registration.
      ,
      • Eiland R.B.
      • Maare C.
      • Sjöström D.
      • Samsøe E.
      • Behrens C.F.
      Dosimetric and geometric evaluation of the use of deformable image registration in adaptive intensity-modulated radiotherapy for head-and-neck cancer.
      ,
      • He Y.
      • Cazoulat G.
      • Wu C.
      • Peterson C.
      • McCulloch M.
      • Anderson B.
      • et al.
      Geometric and dosimetric accuracy of deformable image registration between average-intensity images for 4DCT-based adaptive radiotherapy for non-small cell lung cancer.
      ,
      • Masi M.
      • Landoni V.
      • Faiella A.
      • Farneti A.
      • Marzi S.
      • Guerrisi M.
      • et al.
      Comparison of rigid and deformable coregistration between mpMRI and CT images in radiotherapy of prostate bed cancer recurrence.
      ,
      • Zhang L.
      • Wang Z.
      • Shi C.
      • Long T.
      • Xu X.G.
      The impact of robustness of deformable image registration on contour propagation and dose accumulation for head and neck adaptive radiotherapy.
      ]. Finally, singular methods, such as calculating the ratio of the entire dose-volume histogram [
      • Hoon Jung S.
      • Min Yoon S.
      • Ho Park S.
      • Cho B.
      • Won Park J.
      • Jung J.
      • et al.
      Four-dimensional dose evaluation using deformable image registration in radiotherapy for liver cancer.
      ] and the gamma index [
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ] or measuring the impact of dose on PET/CT perfusion [
      • Hardcastle N.
      • Hofman M.S.
      • Hicks R.J.
      • Callahan J.
      • Kron T.
      • MacManus M.P.
      • et al.
      Accuracy and Utility of Deformable Image Registration in 68Ga 4D PET/CT Assessment of Pulmonary Perfusion Changes During and After Lung Radiation Therapy.
      ], have sometimes been developed. A method has also been developed specifically to quantifiy dosimetric uncertainty in DIR based dose accumulation [
      • Kainz K.
      • Garcia Alvarez J.
      • Zhong H.
      • Lim S.
      • Ahunbay E.
      • Tai A.
      • et al.
      Use of a DVH overlay technique for quality assurance of deformable image registration-based dose accumulation.
      ]. Thresholds to define whether the dosimetric impact of the DIR was negligible or not (e.g., V95% that remains above 98% for the deformed PTV, or differences <2.5% between ground-truth and deformed volume metrics) were used in four papers [
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ,
      • Elmahdy M.S.
      • Jagt T.
      • Zinkstok R.T.
      • Qiao Y.
      • Shahzad R.
      • Sokooti H.
      • et al.
      Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer.
      ,

      Qiao Y, Jagt T, Hoogeman M, Lelieveldt BPF, Staring M. Evaluation of an Open Source Registration Package for Automatic Contour Propagation in Online Adaptive Intensity-Modulated Proton Therapy of Prostate Cancer. Front Oncol 2019;9. 10.3389/fonc.2019.01297.

      ,
      • Hvid C.A.
      • Elstrøm U.V.
      • Jensen K.
      • Alber M.
      • Grau C.
      Accuracy of software-assisted contour propagation from planning CT to cone beam CT in head and neck radiotherapy.
      ].
      Table 3Application and dose metrics used in the 32 articles in wich a dosimetric evaluation is performed, sorted by localization and year of publication.
      Article (First author, year, reference)Main localizationApplicationDose metrics
      Thor2013
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ProstateContour propagationRectum and bladder: V70Gy, Dmean and gEUD
      Thor2014
      • Thor M.
      • Andersen E.S.
      • Petersen J.B.B.
      • Sørensen T.S.
      • Noe K.Ø.
      • Tanderup K.
      • et al.
      Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy.
      ProstateDose accumulation/dose deformationBladder: D98%, D66%, D33%, D20%, D2%, Dmean, V40Gy, V70Gy, gEUD, NTCP – Prostate: Dmin, D99,5%
      Saito2018
      • Saito M.
      • Shibata Y.
      • Sano N.
      • Kuriyama K.
      • Komiyama T.
      • Marino K.
      • et al.
      Evaluation of Deformable Image Registration and Dose Accumulation Using Histogram Matching Algorithm between kVCT and MVCT with Helical Tomotherapy.
      ProstateDose accumulation/dose deformationProstate: Dmax, Dmean, D50% – Rectum: V20Gy, V60Gy, V70Gy – Bladder: V60Gy, V70Gy
      Elmahdy2019
      • Elmahdy M.S.
      • Jagt T.
      • Zinkstok R.T.
      • Qiao Y.
      • Shahzad R.
      • Sokooti H.
      • et al.
      Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer.
      ProstateContour propagationProstate, VS and LN: V95%, V107% – Rectum: Dmean, V45Gy, V60Gy, V75Gy – Bladder: Dmean, V45Gy, V65Gy
      Anaya2019
      • Marin Anaya V.
      • Fairfoul J.
      Assessing the feasibility of adaptive planning for prostate radiotherapy using Smartadapt deformable image registration.
      ProstateContour propagationProstate: V95%, D50%, D98%, D2%, CI – Rectum: V50Gy, V60Gy
      Qiao2019

      Qiao Y, Jagt T, Hoogeman M, Lelieveldt BPF, Staring M. Evaluation of an Open Source Registration Package for Automatic Contour Propagation in Online Adaptive Intensity-Modulated Proton Therapy of Prostate Cancer. Front Oncol 2019;9. 10.3389/fonc.2019.01297.

      ProstateContour propagationProstate, VS and LN: V95%, V107% – Rectum: Dmean, V45Gy, V60Gy, V75Gy – Bladder: Dmean, V45Gy, V75Gy
      Masi2021
      • Masi M.
      • Landoni V.
      • Faiella A.
      • Farneti A.
      • Marzi S.
      • Guerrisi M.
      • et al.
      Comparison of rigid and deformable coregistration between mpMRI and CT images in radiotherapy of prostate bed cancer recurrence.
      ProstateContour propagationProstate bed recurrence: V90%, V95%, V100%
      Vozzo2021
      • Vozzo M.
      • Poder J.
      • Yuen J.
      • Bucci J.
      • Haworth A.
      Use of deformable image registration techniques to estimate dose to organs at risk following prostate external beam radiation therapy and high-dose-rate brachytherapy.
      ProstateContour propagationRectum: D1cc, D2cc
      Kainz2022
      • Kainz K.
      • Garcia Alvarez J.
      • Zhong H.
      • Lim S.
      • Ahunbay E.
      • Tai A.
      • et al.
      Use of a DVH overlay technique for quality assurance of deformable image registration-based dose accumulation.
      ProstateDose accumulation/dose deformationProstate, rectum, bladder: deformation vector histograms
      Eiland2014
      • Eiland R.B.
      • Maare C.
      • Sjöström D.
      • Samsøe E.
      • Behrens C.F.
      Dosimetric and geometric evaluation of the use of deformable image registration in adaptive intensity-modulated radiotherapy for head-and-neck cancer.
      H&NContour propagationCTV: D50%, D98%, D2%, other – SC and PG: Dmax, Dmean – Normal tissue: other
      Garcia-Molla2015
      • García-Mollá R.
      • de Marco-Blancas N.
      • Bonaque J.
      • Vidueira L.
      • López-Tarjuelo J.
      • Perez-Calatayud J.
      Validation of a deformable image registration produced by a commercial treatment planning system in head and neck.
      H&NContour propagationBS, brain, larynx, jaw, SC, ON, eyes, PG, CTV: D98%, D95%, Dmean, D50%, D2%
      Rigaud2015
      • Rigaud B.
      • Simon A.
      • Castelli J.
      • Gobeli M.
      • Ospina Arango J.-D.
      • Cazoulat G.
      • et al.
      Evaluation of Deformable Image Registration Methods for Dose Monitoring in Head and Neck Radiotherapy.
      H&NDose accumulation/dose deformationLandmarks: dose error; PG: Dmean
      Hvid2016
      • Hvid C.A.
      • Elstrøm U.V.
      • Jensen K.
      • Alber M.
      • Grau C.
      Accuracy of software-assisted contour propagation from planning CT to cone beam CT in head and neck radiotherapy.
      H&NContour propagationTongue, PG, PCM, SC, SMG, larynx, pharynx, esophagus: Dmean
      Yeap2017
      • Yeap P.L.
      • Noble D.J.
      • Harrison K.
      • Bates A.M.
      • Burnet N.G.
      • Jena R.
      • et al.
      Automatic contour propagation using deformable image registration to determine delivered dose to spinal cord in head-and-neck cancer radiotherapy.
      H&NContour propagationSC: D2%
      Nobnop2017
      • Nobnop W.
      • Chitapanarux I.
      • Neamin H.
      • Wanwilairat S.
      • Lorvidhaya V.
      • Sanghangthum T.
      Evaluation of Deformable Image Registration (DIR) Methods for Dose Accumulation in Nasopharyngeal Cancer Patients during Radiotherapy.
      H&NDose accumulation/dose deformationGTV/CTV: D50%, D2%, D98% – PG: Dmean, D50% – SC: D2%
      Zhang2018
      • Zhang L.
      • Wang Z.
      • Shi C.
      • Long T.
      • Xu X.G.
      The impact of robustness of deformable image registration on contour propagation and dose accumulation for head and neck adaptive radiotherapy.
      H&NDose accumulation/dose deformationGTV: Dmax, Dmin, Dmean, D95%, HI – PG, SC, BS, TL, lens, ON, cochlea: Dmax, Dmean
      Nash2022
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      H&NContour propagationSC, BS: D1cc, ICC- PG, larynx: Dmean, ICC
      Hardcastle2015
      • Hardcastle N.
      • Hofman M.S.
      • Hicks R.J.
      • Callahan J.
      • Kron T.
      • MacManus M.P.
      • et al.
      Accuracy and Utility of Deformable Image Registration in 68Ga 4D PET/CT Assessment of Pulmonary Perfusion Changes During and After Lung Radiation Therapy.
      ThoraxContour propagationOther
      Moriya2017
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ThoraxDose deformationGTV: Dmean, gEUD
      Azcona2019
      • Azcona J.D.
      • Huesa-Berral C.
      • Moreno-Jiménez M.
      • Barbés B.
      • Aristu J.J.
      • Burguete J.
      A novel concept to include uncertainties in the evaluation of stereotactic body radiation therapy after 4D dose accumulation using deformable image registration.
      ThoraxDose accumulation/dose deformationGTV/PTV: Dmin, D95% – Lungs: Dmean, V5Gy, V10Gy, V20Gy – SC, esophagus, heart, ribs: Dmax
      Sarudis2019
      • Sarudis S.
      • Karlsson A.
      • Bibac D.
      • Nyman J.
      • Bäck A.
      Evaluation of deformable image registration accuracy for CT images of the thorax region.
      ThoraxContour propagationGTV: Dmean, D98% – Lung: V12Gy, V20Gy – Heart, SC: D2%
      He2021
      • He Y.
      • Cazoulat G.
      • Wu C.
      • Peterson C.
      • McCulloch M.
      • Anderson B.
      • et al.
      Geometric and dosimetric accuracy of deformable image registration between average-intensity images for 4DCT-based adaptive radiotherapy for non-small cell lung cancer.
      ThoraxDose deformationGTV: HI, other metrics
      Metrics specific to the issue addressed in the article.
      – Lung: V20, other metrics
      Metrics specific to the issue addressed in the article.
      Nenoff2021
      • Nenoff L.
      • Matter M.
      • Amaya E.J.
      • Josipovic M.
      • Knopf A.-C.
      • Lomax A.J.
      • et al.
      Dosimetric influence of deformable image registration uncertainties on propagated structures for online daily adaptive proton therapy of lung cancer patients.
      ThoraxContour propagationCTV: V95%, D2% – Lungs: V20Gy
      Jung2013
      • Hoon Jung S.
      • Min Yoon S.
      • Ho Park S.
      • Cho B.
      • Won Park J.
      • Jung J.
      • et al.
      Four-dimensional dose evaluation using deformable image registration in radiotherapy for liver cancer.
      AbdomenDose accumulation/dose deformationGTV/PTV: Dmax, Dmean – Liver, duodenum, stomach, kidneys: Dmax, Dmean, D5cc, gEUD, NTCP
      Lee2018
      • Lee S.
      • Kim H.
      • Ji Y.
      • Cho B.
      • Kim S.S.
      • Jung J.
      • et al.
      Evaluation of Hepatic Toxicity after Repeated Stereotactic Body Radiation Therapy for Recurrent Hepatocellular Carcinoma using Deformable Image Registration.
      AbdomenDose accumulation/dose deformationLiver: Dmax, Dmean, V5-10-15-20-25-30-40-50-60-75-90-105-120Gy
      Kubota2019
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      AbdomenContour propagationGTV/CTV: V95%, – Stomach: V50%, V10% – Duodenum: V50%, V10%
      Abe2014
      • Abe T.
      • Tamaki T.
      • Makino S.
      • Ebara T.
      • Hirai R.
      • Miyaura K.
      • et al.
      Assessing cumulative dose distributions in combined radiotherapy for cervical cancer using deformable image registration with pre-imaging preparations.
      CervixDose accumulation/dose deformationCTV: D90% – Rectum, bladder: D2cc
      Kadoya2017
      • Kadoya N.
      • Miyasaka Y.
      • Yamamoto T.
      • Kuroda Y.
      • Ito K.
      • Chiba M.
      • et al.
      Evaluation of rectum and bladder dose accumulation from external beam radiotherapy and brachytherapy for cervical cancer using two different deformable image registration techniques.
      CervixDose accumulation/dose deformationRectum, bladder: D0.1cc, D1cc, D2cc
      Chapmaan2018
      • Chapman C.H.
      • Polan D.
      • Vineberg K.
      • Jolly S.
      • Maturen K.E.
      • Brock K.K.
      • et al.
      Deformable image registration–based contour propagation yields clinically acceptable plans for MRI-based cervical cancer brachytherapy planning.
      CervixContour propagationCTV: D90% – Rectum, bladder, sigmoid: D2cc, other
      Mohamadi2019
      • Mohammadi R.
      • Mahdavi S.R.
      • Jaberi R.
      • Siavashpour Z.
      • Janani L.
      • Meigooni A.S.
      • et al.
      Evaluation of deformable image registration algorithm for determination of accumulated dose for brachytherapy of cervical cancer patients.
      CervixDose accumulation/dose deformationRectum, bladder: D0.1cc, D1cc, D2cc, D5cc
      Zeng2020

      Zeng J, Chen J, Zhang D, Meng M, Zhang B, Qu P, et al. Assessing cumulative dose distributions in combined external beam radiotherapy and intracavitary brachytherapy for cervical cancer by treatment planning based on deformable image registration. Translational Cancer Research 2020;9. 10.21037/tcr-20-1196.

      CervixDose accumulation/dose deformationGTV/CTV/parametrium: D100Gy, D90Gy, V100%
      Lee2019
      • Lee S.
      • Stewart J.
      • Lee Y.
      • Myrehaug S.
      • Sahgal A.
      • Ruschin M.
      • et al.
      Improved dosimetric accuracy with semi-automatic contour propagation of organs-at-risk in glioblastoma patients undergoing chemoradiation.
      BrainContour propagationBS, ON, eyes, chiasm: D0.03cc
      BS: brainstem; CI: conformity index; CTV: clinical target volume; Dmax: max dose; Dmean: mean dose; Dmin: min dose; gEUD: generalized equivalent uniform dose; GTV: gross tumor volume; HI: homogeneity index; ICC: intraclass correlation; LN: lymph nodes; NTCP: normal tissue complication probability; ON: optic nerves; PCM: pharyngeal constrictor muscle; PG: parotid glands; SC: spinal canal; SMG: submandibular glands; TL: temporal lobe.
      It can be seen that a variety of methods are used to quantify the dosimetric impact of DIR errors. To date, it is hard to say that some approaches have emerged as methods sufficiently robust for practical clinical use [
      • Rong Y.i.
      • Rosu-Bubulac M.
      • Benedict S.H.
      • Cui Y.
      • Ruo R.
      • Connell T.
      • et al.
      Rigid and Deformable Image Registration for Radiation Therapy: A Self-Study Evaluation Guide for NRG Oncology Clinical Trial Participation.
      ].

      Use of threshold values for metrics

      Tolerance thresholds for metrics quantifying the quality of a deformable registration can theoretically be used to define a value above which the registration is acceptable. However, their use is far from systematic in the literature, and shortcomings included the absence of tolerance levels, the use of a quantitative tolerance threshold, or a subjective QR represented by 69%, 22% and 7%, respectively, of the 107 reviewed articles.
      The QR allows for a subjective assessment of the clinical acceptability of the registration and is complementary to a quantitative procedure [
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      ]. However, its use remains limited, perhaps because of its time-consuming nature and the need for expert evaluators. In practice, an expert clinician evaluates the acceptability of the contours resulting from the registration according to classification with three, four, or five tolerance levels. In our review, the use of 3-, 4- and 5-level classifications was retrieved in two, five and one article, respectively. The most frequent levels were i/acceptable contour, ii/minor deviations, iii/major deviations [
      • Hardcastle N.
      • van Elmpt W.
      • De Ruysscher D.
      • Bzdusek K.
      • Tomé W.A.
      Accuracy of deformable image registration for contour propagation in adaptive lung radiotherapy.
      ], or i/good, ii/acceptable, iii/need of adjustment, iv/poor [
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      ]. An analysis of practices by year of publication showed that QR was mostly used until 2015 (Fig. 10).
      Quantitative tolerance thresholds were used in 24/107 papers, showing that their use was not widespread. There was greater use of thresholds after 2018, shortly after the publication of TG132 [

      Brock KK, Mutic S, McNutt TR, Li H, Kessler ML. Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the AAPM Radiation Therapy Committee Task Group No. 132. Med Phys 2017;44:e43–76. 10.1002/mp.12256.

      ]. This publication provides tolerance levels for the following five metrics: TRE, MDA, DSC, JACO and IC (Appendix A). On Fig. 10, it is observed that quantitative thresholds are used in 25 to 60% of the published articles after 2018, whereas they were almost never used before. Remarkably, the existence of these levels may also have encouraged the use of some metrics rather than others, as was pointed out above for the MDA (Fig. 9).
      In general, we observe that these thresholds were often not used to quantify the quality of a DIR (69% of the 107 articles did not use thresholds). This was justified by their methodological inadequacy, as when the author compared the performances of different registration algorithms, where the only value of the metric allows determining the best algorithm [
      • Mohamed A.S.R.
      • Ruangskul M.-N.
      • Awan M.J.
      • Baron C.A.
      • Kalpathy-Cramer J.
      • Castillo R.
      • et al.
      Quality Assurance Assessment of Diagnostic and Radiation Therapy-Simulation CT Image Registration for Head and Neck Radiation Therapy: Anatomic Region of Interest–based Comparison of Rigid and Deformable Algorithms.
      ]. However, this reason alone cannot explain why these thresholds remained underutilized. Reaching the threshold tolerance does not necessarily foreshadow the clinical relevance of a DIR, in particular for overlap-based metrics such as DSC or JAC, which was another explanation [
      • Kumarasiri A.
      • Siddiqui F.
      • Liu C.
      • Yechieli R.
      • Shah M.
      • Pradhan D.
      • et al.
      Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
      ,
      • Loi G.
      • Fusella M.
      • Lanzi E.
      • Cagni E.
      • Garibaldi C.
      • Iacoviello G.
      • et al.
      Performance of commercially available deformable image registration platforms for contour propagation using patient-based computational phantoms: A multi-institutional study.
      ]. Indeed, DSC can provide a false impression of good DIR, as, on the one hand, it penalizes small organs, but on the other hand, it is too permissive for large organs [
      • Scaggion A.
      • Fiandra C.
      • Loi G.
      • Vecchi C.
      • Fusella M.
      Free-to-use DIR solutions in radiotherapy: Benchmark against commercial platforms through a contour-propagation study.
      ]. For this category of metric, Sherer et al. advised against a universal cut-off to indicate the acceptability of specific contour due to the challenge of the variation of cut-offs by organs [
      • Sherer M.V.
      • Lin D.
      • Elguindi S.
      • Duke S.
      • Tan L.-T.
      • Cacicedo J.
      • et al.
      Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: a critical review.
      ].

      Use of threshold values for metrics in ART

      With several commercial systems available, ART is growing and can be applied at different treatment timescales: offline between fractions, online immediately prior to a fraction and real-time during the fraction. One advantage of offline ART is that in-room imaging such as CBCT, helical kVCT or MRI may be used, or conventional CT or MRI on a simulator or diagnostic scanner if sufficient image quality cannot be achieved with in-room imaging [
      • Green O.L.
      • Henke L.E.
      • Hugo G.D.
      Practical Clinical Workflows for Online and Offline Adaptive Radiation Therapy.
      ]. For online ART, DIR is employed for contour propagation, plan adjustment, electron density mapping or dose accumulation. Any error in the image deformation will affect the ART workflow which has to be done in a very short time. A summary of the minimum requirements to consider for ART workflow components was proposed [
      • Glide-Hurst C.K.
      • Lee P.
      • Yock A.D.
      • Olsen J.R.
      • Cao M.
      • Siddiqui F.
      • et al.
      Adaptive Radiation Therapy (ART) Strategies and Technical Considerations: A State of the ART Review From NRG Oncology.
      ]: the minimum requirement suggested for DIR evaluation is a visual assessment and a MDA within the magnitude of maximum voxel dimension. For contour propagation, visual assessment and DSC >0.8 are proposed as minimum requirements. However, we have seen that this type of metric and threshold does not necessarily predict the clinical relevance of a DIR. Future developments are therefore needed to improve methods of ART quality assurance, especially in the online ART vendor platform: one example might be the use of the JACO obtained from DIR between planning and daily image, to rapidly determine within one minute the need for online ART [
      • Lim S.N.
      • Ahunbay E.E.
      • Nasief H.
      • Zheng C.
      • Lawton C.
      • Li X.A.
      Indications of Online Adaptive Replanning Based On Organ Deformation.
      ].

      Correlation between metrics

      The choice of metrics for characterizing the performance of a DIR is challenging. The correlation between these metrics is required to eliminate duplicated metrics. By comparing quantitative and qualitative metrics, the clinical value of the most useful metric can be assessed. Finally, in DIR applications involving dose, it may be relevant to know whether a geometric metric reflects the dosimetric impact of a DIR.
      We identified 15/107 papers in which the authors sought to quantify correlations between the following metrics: correlations between quantitative and qualitative metrics, geometric and dosimetric metrics and between different types of geometric metrics (five, five and seven articles, respectively; see Table 4).
      Table 4Details of articles in which correlations between metrics are studied.
      Type of metricsCorrelation studiedNumber of articlesCorrelation: yes or no
      Between quantitative and qualitative metricsDSC/QR5Thörnqvist, 2010
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      : yes for rectum, but not possible to relate a certain value of DSC to a clinical score for other OARs

      Thor, 2011
      • Thor M.
      • Petersen J.B.B.
      • Bentzen L.
      • Høyer M.
      • Muren L.P.
      Deformable image registration for contour propagation from CT to cone-beam CT scans in radiotherapy of prostate cancer.
      : yes for bladder, no for other volumes

      Hardcastle, 2012
      • Hardcastle N.
      • Tomé W.A.
      • Cannon D.M.
      • Brouwer C.L.
      • Wittendorp P.W.
      • Dogan N.
      • et al.
      A multi-institution evaluation of deformable image registration algorithms for automatic organ delineation in adaptive head and neck radiotherapy.
      : moderate correlation for OAR, no for targets

      Hardcastle, 2013
      • Hardcastle N.
      • van Elmpt W.
      • De Ruysscher D.
      • Bzdusek K.
      • Tomé W.A.
      Accuracy of deformable image registration for contour propagation in adaptive lung radiotherapy.
      : yes for OARs, no for targets

      Dyer, 2019
      • Dyer B.A.
      • Yuan Z.
      • Qiu J.
      • Benedict S.H.
      • Valicenti R.K.
      • Mayadev J.S.
      • et al.
      Factors associated with deformation accuracy and modes of failure for MRI-optimized cervical brachytherapy using deformable image registration.
      : strong to moderate correlation, depending on the DIR algorithms
      MSHD/QR2Hardcastle, 2012
      • Hardcastle N.
      • Tomé W.A.
      • Cannon D.M.
      • Brouwer C.L.
      • Wittendorp P.W.
      • Dogan N.
      • et al.
      A multi-institution evaluation of deformable image registration algorithms for automatic organ delineation in adaptive head and neck radiotherapy.
      : moderate correlation for OAR, no for GTV

      Hardcastle, 2013
      • Hardcastle N.
      • van Elmpt W.
      • De Ruysscher D.
      • Bzdusek K.
      • Tomé W.A.
      Accuracy of deformable image registration for contour propagation in adaptive lung radiotherapy.
      : no correlation for GTV and OAR
      DTA/QR1Dyer, 2019
      • Dyer B.A.
      • Yuan Z.
      • Qiu J.
      • Benedict S.H.
      • Valicenti R.K.
      • Mayadev J.S.
      • et al.
      Factors associated with deformation accuracy and modes of failure for MRI-optimized cervical brachytherapy using deformable image registration.
      : strong to moderate correlation, depending on the DIR algorithms
      Between geometric and dosimetric metricsDSC/DOSE3Thor, 2013
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      : yes

      Moriya, 2017
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      : moderate correlation

      Sarudis, 2019
      • Sarudis S.
      • Karlsson A.
      • Bibac D.
      • Nyman J.
      • Bäck A.
      Evaluation of deformable image registration accuracy for CT images of the thorax region.
      : no
      COM/DOSE1Sarudis, 2019
      • Sarudis S.
      • Karlsson A.
      • Bibac D.
      • Nyman J.
      • Bäck A.
      Evaluation of deformable image registration accuracy for CT images of the thorax region.
      : no
      VOL/DOSE – QR/DOSE1Thor, 2013
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      : yes for some dose parameters (not all)
      MDA/DOSE1Kubota, 2019
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      : low to moderate correlation
      Between different kind of geometric metricsDSC/MDA1Kubota, 2019
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      : high correlation
      DSC/VOL2Peroni, 2013
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      : no

      Thor, 2014
      • Thor M.
      • Andersen E.S.
      • Petersen J.B.B.
      • Sørensen T.S.
      • Noe K.Ø.
      • Tanderup K.
      • et al.
      Evaluation of an application for intensity-based deformable image registration and dose accumulation in radiotherapy.
      : no, even if a high DSC is associated with high VOL differences for some patients
      DSC/COM – DSC/HD – COM/HD1Kumarasiri, 2014
      • Kumarasiri A.
      • Siddiqui F.
      • Liu C.
      • Yechieli R.
      • Shah M.
      • Pradhan D.
      • et al.
      Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
      : COM/HD correlation is stronger than COM/DSC one that is stronger than DSC/HD one
      DSC/MI – DSC/SCR – MCS/MI – MCS/SCR1Fallone, 2010
      • Fallone B.G.
      • Rivest D.R.C.
      • Riauka T.A.
      • Murtha A.D.
      Assessment of a commercially available automatic deformable registration system.
      : increased SCR and NMI values correspond to improvements in DSC and MCS, but this trend is clearly not universal
      DSC/PPV – DSC/SD – VOL/PPV – VOL/SD – SD/PPV1Peroni, 2013
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      : PPV has more discriminative power than DSC, but yet fails in cases of full inclusion of the contours into another. VOL is simple and immediate, but it lacks in detecting misaligned volumes. VOL combined with SD provides the necessary information
      DDM/TE – DDM/ICE – DDM/DCS – DDM/VOL1Saleh, 2016
      • Saleh Z.
      • Thor M.
      • Apte A.P.
      • Sharp G.
      • Tang X.
      • Veeraraghavan H.
      • et al.
      A multiple-image-based method to evaluate the performance of deformable image registration in the pelvis.
      : DDM resulted in higher correlations with the investigated volume ratios and the DSC compared to TE and ICE
      MCD/JAC – GCD/JAC – MCD/GCD1Stützer, 2016
      • Stützer K.
      • Haase R.
      • Lohaus F.
      • Barczyk S.
      • Exner F.
      • Löck S.
      • et al.
      Evaluation of a deformable registration algorithm for subsequent lung computed tomography imaging during radiochemotherapy.
      : good correlation
      PPV/SENS1Thörnqvist, 2010
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      : better correlation for rectum than for bladder
      COM: center of mass; DDM: distance discordance metric; DOSE: dose-based parameters; DSC: Dice similarity coefficient; DTA: distance to agreement; GCD: geometric center distance; HD: Hausdorff distance; ICE: inverse consistency error; JAC: Jaccard coefficient; MCD: mean contour distance; MCS: mean contour separation; MDA: mean distance to agreement; MI: mutual information; MSHD: mean of the slicewise Hausdorff distances; OARs: organs at risk; PPV: positive predictive value; QR: qualitative rating; SCR: symmetric correlation ratio; SENS: sensitivity; VOL: volume analysis (for example volume comparison).
      * Metrics specific to the issue addressed in the article.
      We first observed that the number of articles was relatively small and that most correlations were studied only once. Second, the limited number of patients or registrations used in these articles did not always allow statistically significant results. In the study of Peroni et al., only three patients were analyzed [
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      ]; in the article of Hardcastle et al., too few GTV samples were used to reach a statistically significant correlation [
      • Hardcastle N.
      • Tomé W.A.
      • Cannon D.M.
      • Brouwer C.L.
      • Wittendorp P.W.
      • Dogan N.
      • et al.
      A multi-institution evaluation of deformable image registration algorithms for automatic organ delineation in adaptive head and neck radiotherapy.
      ]. Finally, the correlation between metrics was sometimes only calculated in a qualitative way. Kumarasiri et al. showed that the correlation between COM and HD was “stronger” than between COM and DSC and “stronger” than between DSC and HD [
      • Kumarasiri A.
      • Siddiqui F.
      • Liu C.
      • Yechieli R.
      • Shah M.
      • Pradhan D.
      • et al.
      Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
      ].
      Importantly, correlations between metrics validated for one type of algorithm or organ were not necessarily validated for others. Dyer et al. showed a strong correlation between DTA and a clinical score for one algorithm, while this correlation was considered moderate for another algorithm [
      • Dyer B.A.
      • Yuan Z.
      • Qiu J.
      • Benedict S.H.
      • Valicenti R.K.
      • Mayadev J.S.
      • et al.
      Factors associated with deformation accuracy and modes of failure for MRI-optimized cervical brachytherapy using deformable image registration.
      ]. The same phenomenon was observed for organs. Indeed, a correlation between the DSC and a clinical score has been demonstrated for the rectum but not for the other OARs [
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      ]. However, formal methods are proposed for selecting the most appropriate metrics according to their correlation. [
      • Juan-Cruz C.
      • Fast M.F.
      • Sonke J.-J.
      A multivariable study of deformable image registration evaluation metrics in 4DCT of thoracic cancer patients.
      ,
      • Taha A.A.
      • Hanbury A.
      Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool.
      ].
      When results of the most studied correlation, i.e., between the DSC and a qualitative scoring, were analyzed, reliability between DSC and the clinical quality of a DIR cannot be generalized. For most targets and studied OARs, there was no evidence that the DSC value reflects a subjective clinical judgment [
      • Thörnqvist S.
      • Petersen J.B.B.
      • Høyer M.
      • Bentzen L.N.
      • Muren L.P.
      Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
      ,
      • Hardcastle N.
      • van Elmpt W.
      • De Ruysscher D.
      • Bzdusek K.
      • Tomé W.A.
      Accuracy of deformable image registration for contour propagation in adaptive lung radiotherapy.
      ,
      • Thor M.
      • Petersen J.B.B.
      • Bentzen L.
      • Høyer M.
      • Muren L.P.
      Deformable image registration for contour propagation from CT to cone-beam CT scans in radiotherapy of prostate cancer.
      ,
      • Hardcastle N.
      • Tomé W.A.
      • Cannon D.M.
      • Brouwer C.L.
      • Wittendorp P.W.
      • Dogan N.
      • et al.
      A multi-institution evaluation of deformable image registration algorithms for automatic organ delineation in adaptive head and neck radiotherapy.
      ]. The same conclusion can be drawn between MSHD and clinical scoring, especially since the correlation was less marked than that for DSC.
      Today, dose accumulation is one of the most frequent applications of DIR in radiotherapy [

      Kadoya N, Kito S, Kurooka M, Saito M, Takemura A, Tohyama N, et al. Factual survey of the clinical use of deformable image registration software for radiotherapy in Japan. Journal of Radiation Research 2019;60:546–53. 10.1093/jrr/rrz034.

      ,
      • Rigaud B.
      • Simon A.
      • Castelli J.
      • Lafond C.
      • Acosta O.
      • Haigron P.
      • et al.
      Deformable image registration for radiation therapy: principle, methods, applications and evaluation.
      ], and a dosimetric evaluation is mandatory to judge the performance of DIR in terms of clinical implementation [
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ]. Consequently, assessing whether geometric metrics quantifying the quality of a DIR indicate its dosimetric impact is relevant. Some papers showed a correlation between dosimetric metrics and DSC, MDA or qualitative metrics [
      • Thor M.
      • Bentzen L.
      • Elstrøm U.V.
      • Petersen J.B.B.
      • Muren L.P.
      Dose/volume-based evaluation of the accuracy of deformable image registration for the rectum and bladder.
      ,
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ,
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      ], and other papers moderate correlation or no correlation at all {Citation}. However, because of the scarcity of the papers and because of the varied comparisons and their diverse results, their conclusions must be interpreted with caution. The correlation between geometric and dosimetric metrics also seems to depend on the volume and the type of metric as follows: in the study of Kubota et al., the correlation between MDA and V95% was moderate for the target, while it was low between MDA and V50Gy for OARs [
      • Kubota Y.
      • Okamoto M.
      • Li Y.
      • Shiba S.
      • Okazaki S.
      • Komatsu S.
      • et al.
      Evaluation of Intensity- and Contour-Based Deformable Image Registration Accuracy in Pancreatic Cancer Patients.
      ]. In current practice as dose accumulation or ART, it seems therefore crucial to use dosimetric metrics rather than geometric metrics to quantify the quality of a DIR.
      In summary, even if significant correlations between metrics intended to quantify the quality of a DIR have been demonstrated in the literature, the level of evidence seems to still be too low to establish robust and reproducible correlations, regardless of the type of metric considered. This conclusion is comparable with a previous review by Sherer et al., which was dedicated to autosegmentation contours [
      • Sherer M.V.
      • Lin D.
      • Elguindi S.
      • Duke S.
      • Tan L.-T.
      • Cacicedo J.
      • et al.
      Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: a critical review.
      ] and stated that geometric indices are not well correlated with clinical endpoints or dosimetric parameters.

      Intra- and interobserver variability in the delineation of ground-truth volumes

      Inter- and intraobserver variability in the delineation of target volumes and OARs is a known source of uncertainty in radiotherapy [
      • Joskowicz L.
      • Cohen D.
      • Caplan N.
      • Sosna J.
      Inter-observer variability of manual contour delineation of structures in CT.
      ,
      • Choi H.J.
      • Kim Y.S.
      • Lee S.H.
      • Lee Y.S.
      • Park G.
      • Jung J.H.
      • et al.
      Inter- and intra-observer variability in contouring of the prostate gland on planning computed tomography and cone beam computed tomography.
      ]. Metrics for quantifying the quality of a DIR based on ground-truth structures are therefore sensitive to this variability.
      To make the use of these metrics relevant, the assessment of DIR quality needs to account for observer variations [
      • Hoffmann C.
      • Krause S.
      • Stoiber E.M.
      • Mohr A.
      • Rieken S.
      • Schramm O.
      • et al.
      Accuracy quantification of a deformable image registration tool applied in a clinical setting.
      ], which was done in 39 articles. Twenty-nine articles studied interobserver variability alone, five studied inter- and intraobserver variability and five studied intraobserver variability alone.

      Interobserver variability

      To account for interobserver variability, contours must be performed by several separate observers. In the 34 papers in which this variability was accounted for, an average of 3.1 observers contoured the structures, with a maximum of 6 and a minimum of 2 observers. To our knowledge, the literature did not indicate the minimum number of observers needed to effectively account for interobserver variability. The observer groups were composed of either a single specialty (mainly radiation oncologists) or combinations of specialties (radiation oncologists, medical oncologists, residents, physicists, dosimetrists, and radiologists). The number of years of experience of the observers varies widely. The volumes of interest were bypassed blinded to other observers. It was rarely specified whether the observers use the same delineation guidelines, rely on departmental protocols, or rely on personal experience. It was never specified whether the observers belong to the same center, which may impact the contours due to different teaching and learning styles.
      We observed the following two types of strategies: the creation of a consensus ground-truth contour or the quantification of interobserver variability that is compared to other uncertainties. The most commonly used method for creating a consensus was simultaneous truth and performance level estimation (STAPLE) [
      • Warfield S.K.
      • Zou K.H.
      • Wells W.M.
      Simultaneous Truth and Performance Level Estimation (STAPLE): An Algorithm for the Validation of Image Segmentation.
      ]. This last method, used in six papers [
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      ,
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      ,
      • Gardner S.J.
      • Wen N.
      • Kim J.
      • Liu C.
      • Pradhan D.
      • Aref I.
      • et al.
      Contouring variability of human- and deformable-generated contours in radiotherapy for prostate cancer.
      ,
      • Liang X.
      • Bibault J.-E.
      • Leroy T.
      • Escande A.
      • Zhao W.
      • Chen Y.
      • et al.
      Automated contour propagation of the prostate from pCT to CBCT images via deep unsupervised learning.
      ,
      • Riegel A.C.
      • Antone J.G.
      • Zhang H.
      • Jain P.
      • Raince J.
      • Rea A.
      • et al.
      Deformable image registration and interobserver variation in contour propagation for radiation therapy planning.
      ,
      • Shaaer A.
      • Davidson M.
      • Semple M.
      • Nicolae A.
      • Mendez L.C.
      • Chung H.
      • et al.
      Clinical evaluation of an MRI-to-ultrasound deformable image registration algorithm for prostate brachytherapy.
      ], was introduced in 2004 and allows the generation of a reference contour from several contours from different observers. The STAPLE algorithm uses a probability map to create a fit based on a set of manual contours from multiple clinicians, allowing us to compensate for violation of contour protocols by experts [
      • Peroni M.
      • Spadea M.F.
      • Riboldi M.
      • Falcone S.
      • Vaccaro C.
      • Sharp G.C.
      • et al.
      Validation of automatic contour propagation for 4D treatment planning using multiple metrics.
      ]. Still in the strategy of creating a ground-truth contour, in six articles, one observer was reviewed by a second observer to reach a consensus [
      • Moriya S.
      • Tachibana H.
      • Kitamura N.
      • Sawant A.
      • Sato M.
      Dose warping performance in deformable image registration in lung.
      ,

      Qiao Y, Jagt T, Hoogeman M, Lelieveldt BPF, Staring M. Evaluation of an Open Source Registration Package for Automatic Contour Propagation in Online Adaptive Intensity-Modulated Proton Therapy of Prostate Cancer. Front Oncol 2019;9. 10.3389/fonc.2019.01297.

      ,
      • Dyer B.A.
      • Yuan Z.
      • Qiu J.
      • Benedict S.H.
      • Valicenti R.K.
      • Mayadev J.S.
      • et al.
      Factors associated with deformation accuracy and modes of failure for MRI-optimized cervical brachytherapy using deformable image registration.
      ,
      • Anderson B.M.
      • Lin Y.-M.
      • Lin E.Y.
      • Cazoulat G.
      • Gupta S.
      • Kyle Jones A.
      • et al.
      A novel use of biomechanical model-based deformable image registration (DIR) for assessing colorectal liver metastases ablation outcomes.
      ,

      Han MC, Kim J, Hong C-S, Chang KH, Han SC, Park K, et al. Performance Evaluation of Deformable Image Registration Algorithms Using Computed Tomography of Multiple Lung Metastases. Technol Cancer Res Treat 2022;21:15330338221078464. 10.1177/15330338221078464.

      ,
      • Omidi A.
      • Weiss E.
      • Wilson J.S.
      • Rosu-Bubulac M.
      Quantitative assessment of intra- and inter-modality deformable image registration of the heart, left ventricle, and thoracic aorta on longitudinal 4D-CT and MR images.
      ]. However, details were missing on how this double check was performed. Other authors choose more singular methods, such as calculating the median of the contours as a reference contour [
      • Faggiano E.
      • Fiorino C.
      • Scalco E.
      • Broggi S.
      • Cattaneo M.
      • Maggiulli E.
      • et al.
      An automatic contour propagation method to follow parotid gland deformation during head-and-neck cancer tomotherapy.
      ]. Jin et al. used an expert consensus [
      • Jin R.
      • Liu Y.
      • Chen M.
      • Zhang S.
      • Song E.
      Contour propagation for lung tumor delineation in 4D-CT using tensor-product surface of uniform and non-uniform closed cubic B-splines.
      ] but did not specify the methodology used.
      Some authors choose to quantify the interobserver variability by computing the variance of each observer on the pairwise distance (PWD-ANOVA) [
      • Mencarelli A.
      • van Beek S.
      • van Kranen S.
      • Rasch C.
      • van Herk M.
      • Sonke J.-J.
      Validation of deformable registration in head and neck cancer using analysis of variance.
      ,
      • Mencarelli A.
      • van Kranen S.R.
      • Hamming-Vrieze O.
      • van Beek S.
      • Nico Rasch C.R.
      • van Herk M.
      • et al.
      Deformable image registration for adaptive radiation therapy of head and neck cancer: accuracy and precision in the presence of tumor changes.
      ]. To quantify the accuracy of the expert contours and the deformable registration algorithm, the PWD-ANOVA method calculates the random errors of the observers by studying the variation of the differences (human or automatic). A limitation of this technique is its sensitivity to outliers. Riegel et al. quantified the interobserver variation by calculating the mean variation between two surfaces [
      • Riegel A.C.
      • Antone J.G.
      • Zhang H.
      • Jain P.
      • Raince J.
      • Rea A.
      • et al.
      Deformable image registration and interobserver variation in contour propagation for radiation therapy planning.
      ] and showed that interobserver variation can be superior to DIR uncertainty. The Pearson correlation coefficient was used in another study to compare the contours performed by six observers and distorted contours in different ways [
      • Gaede S.
      • Olsthoorn J.
      • Louie A.V.
      • Palma D.
      • Yu E.
      • Yaremko B.
      • et al.
      An evaluation of an automated 4D-CT contour propagation tool to define an internal gross tumour volume for lung cancer radiotherapy.
      ]. The Krippendorff alpha reliability coefficient (KALPA) can be used to assess interobserver reliability (low if <0.5, excellent if >0.9) [
      • Mee M.
      • Stewart K.
      • Lathouras M.
      • Truong H.
      • Hargrave C.
      Evaluation of a deformable image registration quality assurance tool for head and neck cancer patients.
      ]. These results showed that interobserver variability depends on anatomical structures, selected geometric metrics and image quality.
      It should be noted that in the papers mentioned above, quantification of interobserver variability is generally not performed on all patients in each study but only on a restricted part of the patients and even only on one [
      • Balik S.
      • Weiss E.
      • Jan N.
      • Roman N.
      • Sleeman W.C.
      • Fatyga M.
      • et al.
      Evaluation of 4-dimensional Computed Tomography to 4-dimensional Cone-Beam Computed Tomography Deformable Image Registration for Lung Cancer Adaptive Radiation Therapy.
      ].

      Intraobserver variability

      Intraobserver variability was taken into account only in ten of the articles [
      • Yeap P.L.
      • Noble D.J.
      • Harrison K.
      • Bates A.M.
      • Burnet N.G.
      • Jena R.
      • et al.
      Automatic contour propagation using deformable image registration to determine delivered dose to spinal cord in head-and-neck cancer radiotherapy.
      ,
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      ,
      • Chapman C.H.
      • Polan D.
      • Vineberg K.
      • Jolly S.
      • Maturen K.E.
      • Brock K.K.
      • et al.
      Deformable image registration–based contour propagation yields clinically acceptable plans for MRI-based cervical cancer brachytherapy planning.
      ,
      • Riegel A.C.
      • Antone J.G.
      • Zhang H.
      • Jain P.
      • Raince J.
      • Rea A.
      • et al.
      Deformable image registration and interobserver variation in contour propagation for radiation therapy planning.
      ,
      • Gaede S.
      • Olsthoorn J.
      • Louie A.V.
      • Palma D.
      • Yu E.
      • Yaremko B.
      • et al.
      An evaluation of an automated 4D-CT contour propagation tool to define an internal gross tumour volume for lung cancer radiotherapy.
      ,
      • Christiansen R.L.
      • Johansen J.
      • Zukauskaite R.
      • Hansen C.R.
      • Bertelsen A.S.
      • Hansen O.
      • et al.
      Accuracy of automatic structure propagation for daily magnetic resonance image-guided head and neck radiotherapy.
      ,
      • Kim J.
      • Kumar S.
      • Liu C.
      • Zhong H.
      • Pradhan D.
      • Shah M.
      • et al.
      A novel approach for establishing benchmark CBCT/CT deformable image registrations in prostate cancer radiotherapy.
      ,
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      ,
      • Speight R.
      • Sykes J.
      • Lindsay R.
      • Franks K.
      • Thwaites D.
      The evaluation of a deformable image registration segmentation technique for semi-automating internal target volume (ITV) production from 4DCT images of lung stereotactic body radiotherapy (SBRT) patients.
      ,
      • Zukauskaite R.
      • Brink C.
      • Hansen C.R.
      • Bertelsen A.
      • Johansen J.
      • Grau C.
      • et al.
      Open source deformable image registration system for treatment planning and recurrence CT scans.
      ]. This variability would be lower than the interobserver variability [
      • Riegel A.C.
      • Antone J.G.
      • Zhang H.
      • Jain P.
      • Raince J.
      • Rea A.
      • et al.
      Deformable image registration and interobserver variation in contour propagation for radiation therapy planning.
      ]; hence, many authors may have chosen to ignore it, even if it can be larger than the one due to DIR [
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      ]. A second contour was performed by the same observer and on the same images with a delay between the two contours to limit memory bias. On average, 6.3 weeks separated the first contour from the second, with the shortest delay being two weeks.
      The methods used varied between articles. For example, the contour made one month apart on the same examination is used as a reference for comparison with the distorted contours through a 2D map of the distribution of variations [
      • Christiansen R.L.
      • Johansen J.
      • Zukauskaite R.
      • Hansen C.R.
      • Bertelsen A.S.
      • Hansen O.
      • et al.
      Accuracy of automatic structure propagation for daily magnetic resonance image-guided head and neck radiotherapy.
      ], showing a relatively low uncertainty of the intraoperator compared to that generated by the deformation. Using the MSD (maximum symmetric distance), another study showed that the combined uncertainty of delineation and deformable registration was doubled compared to the uncertainty of manual delineation alone, from 1.5 to 3 mm [
      • Zukauskaite R.
      • Brink C.
      • Hansen C.R.
      • Bertelsen A.
      • Johansen J.
      • Grau C.
      • et al.
      Open source deformable image registration system for treatment planning and recurrence CT scans.
      ]. Ramadaan et al. assessed intraobserver variability by considering the contours made on the initial dosimetric scan and on the replanning scan [
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      ], including additional uncertainties related to the interpretation of two different images. The dosimetric impact of intraobserver variability has been quantified in another study concerning brachytherapy. The authors did not observe any significant difference in the dosimetric indices calculated for the rectum and sigmoid between manual, deformed and second delineation contours [
      • Chapman C.H.
      • Polan D.
      • Vineberg K.
      • Jolly S.
      • Maturen K.E.
      • Brock K.K.
      • et al.
      Deformable image registration–based contour propagation yields clinically acceptable plans for MRI-based cervical cancer brachytherapy planning.
      ].
      Several influencing factors of intraobserver variability were observed, including the thickness of the slices [
      • Zukauskaite R.
      • Brink C.
      • Hansen C.R.
      • Bertelsen A.
      • Johansen J.
      • Grau C.
      • et al.
      Open source deformable image registration system for treatment planning and recurrence CT scans.
      ], the difficulty in delimiting the cranio-caudal limits of certain structures [
      • Christiansen R.L.
      • Johansen J.
      • Zukauskaite R.
      • Hansen C.R.
      • Bertelsen A.S.
      • Hansen O.
      • et al.
      Accuracy of automatic structure propagation for daily magnetic resonance image-guided head and neck radiotherapy.
      ], the poor image quality [
      • Nash D.
      • Juneja S.
      • Palmer A.L.
      • van Herk M.
      • McWilliam A.
      • Osorio E.V.
      The geometric and dosimetric effect of algorithm choice on propagated contours from CT to cone beam CTs.
      ] and the low contrast of the soft tissues [
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      ]. However, considerable differences between two contours performed by the same operator have sometimes been found, which cannot be explained by these factors alone [
      • Ramadaan I.S.
      • Peick K.
      • Hamilton D.A.
      • Evans J.
      • Iupati D.
      • Nicholson A.
      • et al.
      Validation of Varian’s SmartAdapt® deformable image registration algorithm for clinical application.
      ]: they clearly reflected a difference in interpretation of the two CT images.
      In summary, inter- or intraobserver variability was not studied in nearly 2/3 of the 107 articles. In the remaining 1/3 of the articles, the authors tried to take into account inter- and/or intraobserver variability to isolate the uncertainty linked solely to the use of a DIR. This was done either by determining a consensus reference contour or by quantifying the intra- and/or interobserver errors. The methodologies used varied, and their effectiveness remains difficult to quantify.

      Conclusion

      As the performance of DIRs improves, their use is becoming more widespread, and the methods for characterizing their performance in clinical practice continue to evolve. We performed a literature review and identified 107 articles in which the performance of a DIR was quantified on real patients using operator-based methods. The key elements of the methodologies used were extracted and synthesized. A focus on the metrics used showed in practice how tolerance thresholds are taken into consideration. The lack of data allowing to rule out the correlations between these metrics was highlighted. We established how intra- and interobserver variabilities are taken into account in the establishment of ground truths. This review should help to implement DIR algorithms in clinical practice by providing pertinent information to quantify their performance in real patients.

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      Appendix A. Tolerance levels of metrics used to characterize a DIR, given in AAPM TG 132

      Tabled 1
      Quantitative metricsTolerance
      TREmaximum voxel dimension (∼2–3 mm)
      MDAwithin the contouring uncertainty of the structure or maximum voxel dimension (∼2–3 mm)
      DSCwithin the contouring uncertainty of the structure (∼0,8–0,9)
      Jacobian determinant>0 (0–1 for reasonable volume reduction, >1 for reasonable volume expansion)
      Consistencymaximum voxel dimension (2–3 mm)

      References

        • Yuen J.
        • Barber J.
        • Ralston A.
        • Gray A.
        • Walker A.
        • Hardcastle N.
        • et al.
        An international survey on the clinical use of rigid and deformable image registration in radiotherapy.
        Journal of Applied Clinical Medical Physics. 2020; 21: 10-24
        • Paganelli C.
        • Meschini G.
        • Molinelli S.
        • Riboldi M.
        • Baroni G.
        Patient-specific validation of deformable image registration in radiation therapy: Overview and caveats.
        Med Phys. 2018; 45 (e922): e908https://doi.org/10.1002/mp.13162
      1. Kadoya N, Kito S, Kurooka M, Saito M, Takemura A, Tohyama N, et al. Factual survey of the clinical use of deformable image registration software for radiotherapy in Japan. Journal of Radiation Research 2019;60:546–53. 10.1093/jrr/rrz034.

        • Rigaud B.
        • Simon A.
        • Castelli J.
        • Lafond C.
        • Acosta O.
        • Haigron P.
        • et al.
        Deformable image registration for radiation therapy: principle, methods, applications and evaluation.
        Acta Oncol. 2019; 58: 1225-1237
        • Brock K.K.
        • Mutic S.
        • McNutt T.R.
        • Li H.
        • Kessler M.L.
        Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the AAPM Radiation Therapy Committee Task Group No. 132.
        Med Phys. 2017; 44: e43-e76
        • Akbarzadeh A.
        • Gutierrez D.
        • Baskin A.
        • Ay M.R.
        • Ahmadian A.
        • Alam N.R.
        • et al.
        Evaluation of whole-body MR to CT deformable image registration.
        Journal of Applied Clinical Medical Physics. 2013; 14: 238-253
        • Alam F.
        • Rahman S.U.
        • Khalil A.
        • Khusro S.
        Sajjad M.
        A Review Based on Performance Comparison, Deformable Registration Methods for Medical Images2016: 20
        • Viergever M.A.
        • Maintz J.B.A.
        • Klein S.
        • Murphy K.
        • Staring M.
        • Pluim J.P.W.
        A survey of medical image registration – under review.
        Med Image Anal. 2016; 33: 140-144https://doi.org/10.1016/j.media.2016.06.030
        • Thörnqvist S.
        • Petersen J.B.B.
        • Høyer M.
        • Bentzen L.N.
        • Muren L.P.
        Propagation of target and organ at risk contours in radiotherapy of prostate cancer using deformable image registration.
        Acta Oncol. 2010; 49: 1023-1032https://doi.org/10.3109/0284186X.2010.503662
        • Rong Y.i.
        • Rosu-Bubulac M.
        • Benedict S.H.
        • Cui Y.
        • Ruo R.
        • Connell T.
        • et al.
        Rigid and Deformable Image Registration for Radiation Therapy: A Self-Study Evaluation Guide for NRG Oncology Clinical Trial Participation.
        Practical Radiation Oncology. 2021; 11: 282-298
        • Juan-Cruz C.
        • Fast M.F.
        • Sonke J.-J.
        A multivariable study of deformable image registration evaluation metrics in 4DCT of thoracic cancer patients.
        Phys Med Biol. 2021; 66: 035019
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement.
        PLoS Med. 2009; 6: e1000097
        • Mateo S.
        Procédure pour conduire avec succès une revue de littérature selon la méthode PRISMA.
        Kinésithérapie, la Revue. 2020; 20: 29-37https://doi.org/10.1016/j.kine.2020.05.019
      2. Zotero n.d.

        • Yang D.
        • Brame S.
        • El Naqa I.
        • Aditya A.
        • Wu Y.
        • Goddu S.M.
        • et al.
        Technical note: DIRART–A software suite for deformable image registration and adaptive radiotherapy research.
        Med Phys. 2011; 38: 67-77https://doi.org/10.1118/1.3521468
        • Fu Y.
        • Lei Y.
        • Wang T.
        • Curran W.J.
        • Liu T.
        • Yang X.
        Deep learning in medical image registration: a review.
        Phys Med Biol. 2020;65:20TR01.; 65: 20TR01
        • Xiao H.
        • Ren G.
        • Cai J.
        A review on 3D deformable image registration and its application in dose warping.
        Radiation Medicine and Protection. 2020; 1: 171-178https://doi.org/10.1016/j.radmp.2020.11.002
        • Taha A.A.
        • Hanbury A.
        Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool.
        BMC Med Imaging. 2015; 15: 29https://doi.org/10.1186/s12880-015-0068-x
        • Kumarasiri A.
        • Siddiqui F.
        • Liu C.
        • Yechieli R.
        • Shah M.
        • Pradhan D.
        • et al.
        Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.
        Med Phys. 2014; 41: 121712
        • Sherer M.V.
        • Lin D.
        • Elguindi S.
        • Duke S.
        • Tan L.-T.