Advertisement

The EU medical device regulation: Implications for artificial intelligence-based medical device software in medical physics

Published:March 01, 2021DOI:https://doi.org/10.1016/j.ejmp.2021.02.011

      Highlights

      • Medical device manufacturers apply artificial intelligence (AI) to innovate their products.
      • The European Medical Device Regulation (EU MDR) imposes stringent requirements on medical devices.
      • Under the EU MDR, one core requirement is the application of risk management.
      • Qualified medical physicist experts play a key role in the safety and performance assessment of such tools.
      • It is of paramount importance that the MPEs become familiar with the pillars of AI and EU MDR knowledge.

      Abstract

      Medical device manufacturers are increasingly applying artificial intelligence (AI) to innovate their products and to improve patient outcomes. Health institutions are also developing their own algorithms, to address specific needs for which no commercial product exists.
      Although AI-based algorithms offer good prospects for improving patient outcomes, their wide adoption in clinical practice is still limited. The most significant barriers to the trust required for wider implementation are safety and clinical performance assurance .
      Qualified medical physicist experts (MPEs) play a key role in safety and performance assessment of such tools, before and during integration in clinical practice. As AI methods drive clinical decision-making, their quality should be assured and tested. Occasionally, an MPE may be also involved in the in-house development of such an AI algorithm. It is therefore important for MPEs to be well informed about the current regulatory framework for Medical Devices.
      The new European Medical Device Regulation (EU MDR), with date of application set for 26 of May 2021, imposes stringent requirements that need to be met before such tools can be applied in clinical practice.
      The objective of this paper is to give MPEs perspective on how the EU MDR affects the development of AI-based medical device software. We present our perspective regarding how to implement a regulatory roadmap, from early-stage consideration through design and development, regulatory submission, and post-market surveillance. We have further included an explanation of how to set up a compliant quality management system to ensure reliable and consistent product quality, safety, and performance .

      Abbreviations:

      AI (Artificial Intelligence), CEP (Clinical Evaluation Plan), CER (Clinical Evaluation Report), GDPR (General Data Protection Regulation), GSPR (General Safety and Performance Requirements), IMDRF (International Medical Device Regulators Forum), MDD (Medical Device Directive 93/42/EEC), MDR (Medical Devices Regulation EU 2017/745), MDSW (Medical Device Software), MPE (Medical Physicist Expert), PMCF (Post Market Clinical Follow-up), PMS (Post Market Surveillance), QA (Quality Assurance), QMS (Quality Management System), UDI (Unique Device Identifier), V&V (Verification and Validation)

      Keywords

      1. Introduction

      Of all available health data, medical images – such as those obtained through x-rays, computed tomography, magnetic resonance, nuclear medicine, and ultrasound examinations – possibly constitute the most interesting ones for artificial intelligence (AI) applications. Artificial intelligence is expected to soon play a key role in diagnostic imaging [
      • Nensa F.
      • Demircioglu A.
      • Rischpler C.
      Artificial intelligence in nuclear medicine.
      ,
      • Jha S.
      • Cook T.
      Artificial intelligence in radiology-the state of the future.
      ], radiation therapy [
      • Thompson R.F.
      • Valdes G.
      • Fuller C.D.
      • Carpenter C.M.
      • Morin O.
      • Aneja S.
      • et al.
      Artificial intelligence in radiation oncology: a specialty-wide disruptive transformation?.
      ,
      • Sahiner B.
      • Pezeshk A.
      • Hadjiiski L.M.
      • Wang X.
      • Drukker K.
      • Cha K.H.
      • et al.
      Deep learning in medical imaging and radiation therapy.
      ] and medical physics in general. It has already been demonstrated to successfully improve image quality [
      • Nensa F.
      • Demircioglu A.
      • Rischpler C.
      Artificial intelligence in nuclear medicine.
      ], decrease radiation dosage [
      • Liu P.
      • Wang M.
      • Wang Y.
      • Yu M.
      • Wang Y.
      • Liu Z.
      • et al.
      Impact of deep learning-based optimization algorithm on image quality of low-dose coronary CT angiography with noise reduction: a prospective study.
      ,
      • McCollough C.H.
      • Leng S.
      Use of artificial intelligence in computed tomography dose optimisation.
      ], assign label types and identify pathology locations [
      • McKinney S.M.
      • Sieniek M.
      • Godbole V.
      • Godwin J.
      • Antropova N.
      • Ashrafian H.
      • et al.
      International evaluation of an AI system for breast cancer screening.
      ,
      • Geras K.J.
      • Mann R.M.
      • Moy L.
      Artificial intelligence for mammography and digital breast tomosynthesis: current concepts and future perspectives.
      ,
      • Summers R.M.
      • Beaulieu C.F.
      • Pusanik L.M.
      • Malley J.D.
      • Jeffrey R.B.
      • Glazer D.I.
      • et al.
      Automated polyp detector for CT colonography: feasibility study.
      ,

      Sohee Park, Sang Min Lee, Kyung Hee Lee, Kyu-Hwan Jung, Woong Bae, Jooae Choe Joon Beom Seo. Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings. Eur Radiol. 2020 Mar;30(3):1359–68. 10.1007/s00330-019-06532-x. Epub 2019 Nov 20.

      ,
      • Sim Y.
      • Chung M.J.
      • Kotter E.
      • Yune S.
      • Kim M.
      • Do S.
      • et al.
      Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs.
      ,
      • Tandel G.S.
      • Balestrieri A.
      • Jujaray T.
      • Khanna N.N.
      • Saba L.
      • Suri J.S.
      Multiclass magnetic resonance imaging brain tumor classification using artificial intelligence paradigm.
      ,
      • Hu W.
      • Yang H.
      • Xu H.
      • Mao Y.
      Radiomics based on artificial intelligence in liver diseases: where we are?.
      ], create and optimize protocols [

      Calculating the target exposure index using a deep convolutional neural network and a rule base. Phys Med. 2020 Mar;71:108–14. 10.1016/j.ejmp.2020.02.012. Epub 2020 Feb 27.

      ], accurately segment pathology areas and organs [
      • Pagnozzi A.M.
      • Fripp J.
      • Rose S.E.
      Quantifying deep grey matter atrophy using automated segmentation approaches: a systematic review of structural MRI studies.
      ,

      Simple low-cost approaches to semantic segmentation in radiation therapy planning for prostate cancer using deep learning with non-contrast planning CT images. Phys Med. 2020 Oct;78:93–100. 10.1016/j.ejmp.2020.09.004. Epub 2020 Sep 17.

      ], and optimize technology utilization [
      • Lakhani P.
      • Prater A.B.
      • Hutson R.K.
      • Andriole K.P.
      • Dreyer K.J.
      • Morey J.
      • et al.
      Machine learning in radiology: applications beyond image interpretation.
      ].
      Medical device manufacturers increasingly apply machine learning to innovate their products, to better assist health care providers and to improve patient outcomes. Aside from the industry, health institutions are also developing their own algorithms, to address specific needs for which no commercial product exists.
      The EU Medical Device Regulation (EU MDR) [

      Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance.). vol. 117. 2017.

      ], replacing the EU Medical Device Directive [

      Council Directive 93/42/EEC of 14 June 1993 concerning medical devices. vol. OJ L. 1993.

      ] as of May 26, 2021, imposes stringent regulatory requirements that need to be met before medical devices, including AI software tools, can be used in clinical practice. Medical device software developed by health institutions for in-house use is, for the first time, also affected by many of the conditions and requirements of the EU MDR.
      As standard of care shifts, the landscape evolves and in turn, influences regulatory requirements. In this context, qualified medical physicist experts (MPEs) are in an ideal position to be consulted by notified bodies to assess clinical performance and safety of new medical devices [
      • Sensakovic W.F.
      • Mahesh M.
      Role of the medical physicist in the health care artificial intelligence revolution.
      ]. This is due to their extensive knowledge of the regulatory aspects of medical physics [

      Council Directive 2013/59/Euratom of 5 December 2013 laying down basic safety standards for protection against the dangers arising from exposure to ionising radiation, and repealing Directives 89/618/Euratom, 90/641/Euratom, 96/29/Euratom, 97/43/Euratom and 2003/122/Euratom n.d. https://eur-lex.europa.eu/eli/dir/2013/59/oj (accessed November 30, 2020).

      ] as well as specific expertise in areas such as informatics, image analysis, and statistics. Moreover, in their role as innovator and protocols/processes optimizer as described in the European Guidelines on Medical Physics Expert 174 [

      EC. RP174 European Guidelines on the Medical Physics Expert. 2014. http://op.europa.eu/en/publication-detail/-/publication/b82ed768-4c50-4c9a-a789-98a3b0df5391 (accessed January 29, 2021).

      ], an MPE frequently drives the development of new Medical Device Software (MDSW) tools, for which knowledge of the EU MDR is needed. Finally, the MPEs should lead quality assurance (QA) programs related to AI in medical imaging and radiation therapy, to help maintain acceptable quality and safety in diagnostics and treatment [

      Council Directive 2013/59/Euratom of 5 December 2013 laying down basic safety standards for protection against the dangers arising from exposure to ionising radiation, and repealing Directives 89/618/Euratom, 90/641/Euratom, 96/29/Euratom, 97/43/Euratom and 2003/122/Euratom n.d. https://eur-lex.europa.eu/eli/dir/2013/59/oj (accessed November 30, 2020).

      ].
      The aim of this paper is to give MPEs a perspective on how the EU MDR affects the development and introduction of AI-based medical device software in the clinical workflow.

      2. Medical device software (MDSW)

      The EU MDR applies to processing software that is intended to provide information for one or more of the purposes listed in the definition of a medical device [

      Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance.). vol. 117. 2017.

      ], such as diagnosis, prevention, monitoring, treatment, or alleviation of a disease. It applies to software that provides a diagnosis or therapy by itself, but also to software that merely provides information intended to inform a medical professional in making the final diagnostic or therapeutic decision.
      The EU MDR applies irrespective of whether such software is an executable program, an interactive website, a web service, a script, or just a simple macro in a spreadsheet. Additionally, qualification as a medical device is independent of whether the processing is simple or complex, of the risk posed by the software to the patient or user, of whether it is used by a healthcare professional or by a layperson, and of the computing platform on which it operates, for as long as it is intended for use with human beings or their data for a medical purpose.
      In addition, software that does not meet the definition of a medical device but that is intended to drive or influence the use of a hardware medical device is also subject to the EU MDR, either as a part of, or as an accessory to the hardware, for example, controller software or calibration software for a mammography machine and its diagnostic monitor. Software that does not perform any type of data processing, such as virtual models or orthopaedic templates, is not subject to the regulation. Nor is software with functionality that is limited to storing, communicating, lossless compression or to looking up records in a database (aka ‘simple search’). As the borderline between what is regulated and what is not, is not always clear-cut, even with the help of the existing guidance [

      mdcg_2019_11_guidance_qualification_classification_software (1).pdf n.d.

      ] qualifying software as a medical device can be a difficult task, with heavy consequences in terms of time, money, and responsibility.
      Although the EU MDR is the most stringent and demanding legislation regarding medical device software (MDSW), developers should be aware of other relevant EU laws and guidance documents supporting their implementation. An overview is given in Table 1. Software intended to process personal data might need to comply to the General Data Protection Regulation (GDPR) [

      Regulation (EU) 2016/ 679 of the European Parliament and of the Council - of 27 April 2016 - on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/ 46/ EC (General Data Protection Regulation). n.d.:88.

      ].
      Table 1Overview of relevant EU legislation applicable to medical device software.
      Medical Devices
      Directive 93/42/EECDirective on Medical Devices (MDD). It will be repealed by the Medical Device Regulation (MDR).
      Regulation (EU) 2017/745MDR: it will be fully applicable from 26 May 2021.
      Regulation (EU) 207/2012Regulation on electronic instructions for use of medical devices.
      MEDDEV 2.7/1 (rev. 4)Guidance on the clinical evaluation of medical devices. Although this guidance is developed under the MDD and not legally binding, it offers detailed assistance and formulates strict requirements for clinical evaluation, matching these of the MDR.
      MDCG 2019-11Guidance on qualification and classification of Medical Device Software (MDSW). Not legally binding.
      MDCG 2019-16Guidance on cybersecurity for medical devices.
      MDCG 2020-1Guidance on clinical evaluation of MDSW. Intended to supplement MEDDEV 2.7/1 (rev. 4) for clinical evaluation of MDSW falling under EU MDR.
      Data protection
      Regulation (EU) 2016/679The General Data Protection Regulation (GDPR) on the protection of natural persons regarding the processing of personal data and on the free movement of such data. The regulation applies since 25 May 2018, repealing the previous Directive (95/46/EC).

      3. Regulatory requirements for MDSW

      As soon as software qualifies as a medical device, it must comply with the set of requirements listed in the EU MDR, referred to as the General Safety and Performance Requirements (GSPRs). This compliance is formally verified before a CE marking can be affixed and the product placed on the European market. The CE marking is not required when the software is developed, used, and maintained only within a health institution, under certain conditions: the software cannot be transferred to another legal entity; the health institution justifies that the target patient group's specific needs cannot be met, or cannot be met at the appropriate level of performance by an equivalent device available on the market. The health institution needs to prepare documentation containing, among other things, design and performance information of the device. It also needs to review experience gained from clinical use of the device and take necessary corrective actions. Other conditions are explained further in the MDR. An overview of the regulatory roadmap for MDSW (both for commercial and in-house use) is given in Fig. 1. In the next paragraphs, we will describe in detail each step of this roadmap for commercial manufacturers. The last section addresses the requirements specific for health institutions.
      Figure thumbnail gr1
      Fig. 1Regulatory roadmap for medical device software. The text in italics indicate shared requirements and obligations for both commercial manufacturers and health institutions.

      3.1 Early-stage considerations

      Defining the intended use of a future device is the starting point for all following decisions, including whether or not it qualifies as a medical device. The intended use covers two aspects: (1) the actual medical purpose, e.g., which disease or injury can be diagnosed, treated, or monitored, and (2) the authorised use, understood as defining the intended users and use environment, target patient population, or body parts. Properly described, the intended purpose will therefore provide: (a) confirmation of, whether the regulation applies, (b) the basis for the classification of the future device into one of four risk classes, as required by Article 51 and (c), the starting point for downstream development activities, clinical evaluation and product documentation.
      Medical device software having its own intended medical purpose will be subject to the new classification Rule 11 of the EU MDR. This rule was introduced to address the consequences of indirect harm resulting from software failing to provide correct information. Rule 11 is applicable to MDSW that provides information used to make decisions with diagnostic or therapeutic purposes, to MDSW intended to monitor physiological processes and to other MDSW that does not perform any of the above purposes. Within the framework of this article, we will not discuss MDSW intended to monitor physiological processes, such as respiration, heart rate or body temperature, as this falls outside the scope of the classical tasks of the medical physicist. The interested reader can refer to some recent literature on the topic [
      • Vanegas E.
      • Igual R.
      • Plaza I.
      Sensing systems for respiration monitoring: a technical systematic review.
      ,
      • Li K.H.C.
      • White F.A.
      • Tipoe T.
      • Liu T.
      • Wong M.C.
      • Jesuthasan A.
      • et al.
      The current state of mobile phone apps for monitoring heart rate, heart rate variability, and atrial fibrillation: narrative review.
      ,
      • Ding X.-R.
      • Clifton D.
      • Ji N.
      • Lovell N.H.
      • Bonato P.
      • Chen W.
      • et al.
      Wearable sensing and telehealth technology with potential applications in the coronavirus pandemic.
      ].
      To better understand the classification of MDSW that provides information used to make decisions with diagnostic or therapeutic purposes, we consider the possible risk classification given in Table 2, as agreed by the International Medical Device Regulators Forum (IMDRF) [

      “Software as a Medical Device”: possible framework for risk categorization and corresponding considerations 2014:30.

      ] – a voluntary group of medical device regulators that aims to accelerate international medical device regulatory harmonization – resulting in 9 risk types (3x3 matrix), shown in Table 2. This classification scheme takes into account a combination of the significance of the information (whether it informs clinicians of diagnostic or treatment options, provides information that drives clinical management or provides the diagnosis by itself) and the criticality of a patient’s disease or condition.
      Table 2MDCG 2019-11 Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR: Classification Guidance on Rule 11.
      State of healthcare situation or patient conditionSignificance of information provided by MDSW to healthcare decision
      High

      Treat or diagnose
      Medium

      Drives clinical management
      Low

      Informs clinical management
      Critical situation or patient conditionClass IIIClass IIbClass IIa
      Serious situation or patient conditionClass IIbClass IIaClass IIa
      Non-serious situation or patient conditionClass IIaClass IIaClass IIa
      For example, MDSW for diagnostic image analysis, such as software for the differentiation between acute ischemic and acute haemorrhagic stroke, combines highly significant information (the information provided by the MDSW is used in treatment) and critical condition (the patient’s condition is life threatening, may require major therapeutic intervention, and is time sensitive), leading to a class III categorization. Another example is a radiation treatment planning system which is intended to aid in treatment delivery by using patient-specific information. This planning system drives clinical management (medium significance of information) by providing enhanced support for the safe and effective use of a medical device to a patient in a critical condition that may be life threatening, therefore it classifies as IIb.
      However, as software is considered an active device, it is worth exploring other rules related to active devices (rules 9, 10, 12 and 13) which might also apply. If MDSW also drives or influences the use of a (hardware) device for a medical purpose, it may not be classified lower than the risk class of that hardware medical device. This is applicable to an intelligent CT scanner with a deep learning image reconstruction application, for example. EU MDR special classification rules might also be applicable to devices specifically intended for recording diagnostic images generated by X-ray radiation, which are classified as class IIa. When multiple rules apply, the strictest rule or sub-rule resulting in higher classification is decisive.
      Class I medical devices have the lowest perceived risks and can be placed on the market by the manufacturer alone, by means of a self-declaration of conformity to the legislation. The subsequent risk class stratification leads to class IIa, class IIb and lastly to class III if death or an irreversible health outcome is possible. For class IIa and higher, the involvement of a notified body is required both for reviewing the product’s conformity and the organization’s conformity to the EU MDR requirements, and to deliver the certificate of conformity and the Quality Management System certificate, respectively.
      For products intended for the market, the route towards regulatory approval and the requirements become stricter as the risk class of the device increases. Understanding these requirements, and their implications for the product and the organization, early in the product lifecycle, is fundamental to developing a successful commercialization strategy. For example, limiting the initial medical indications and claims, may place the product in a lower risk class and shorten the time-to-market. Additional clinical data and insights can be gathered once the product has been CE marked and routinely used in the clinical settings, facilitating a staged approach to intended medical purpose extension.

      3.2 Design and development

      Early in the concept or planning phase, a manufacturer must identify the applicable General Safety and Performance Requirements, outlined in EU MDR Annex I, to which the product and the manufacturer’s processes need to comply, given the intended use. Those requirements are high-level, but can be met, to some extent, by applying (product-specific) standards during the design and development phase of the product. Some of them, called harmonised standards, can be used to demonstrate that the products or organization’s processes comply with the EU MDR. Others, such as IEC 82304-1, while not allowing for presumption of conformity to EU laws, represent the state-of-the art in the concerned field. The list of standards relevant to MDSW at the time of writing is provided in Table 3.
      Table 3List of applicable standards for MDSW.
      Company
      EN ISO 13485Standard for a Quality Management System for the design and manufacture of medical devices
      ISO 27001Information technology; Security techniques; Information security management systems
      Product
      EN ISO 14971Application of risk management to medical devices. Specifies terminology, principles and a process for risk management of medical devices. IEC/TR 80002-1 provides guidance on the application of ISO 14971 to MDSW
      EN 62366Application of usability engineering to medical devices. Specifies a process for a manufacturer to analyse, specify, develop and evaluate the usability of a medical device as it relates to safety.
      EN ISO 14155Good Clinical Practice for clinical investigations with medical devices
      EN 1041Specifies requirements for information to be supplied by manufacturers of medical devices
      EN ISO 15223-1Symbols to be used with medical device labels, labelling and information to be supplied
      Software
      IEC 82304-1Health software — Part 1: General requirements for product safety. Applies to the safety and security of health software products designed to operate on general computing platforms and intended to be placed on the market without dedicated hardware. It covers the entire lifecycle including design, development, validation, installation, maintenance, and disposal of health software products.
      EN 62304Medical device software — Software life cycle processes. Defines the life cycle requirements for medical device software. The set of processes, activities, and tasks described in this standard establishes a common framework for medical device software life cycle processes.
      One of the core activities during design and development of the product is risk management. Risk is expressed as the combination of the probability of occurrence of harm (the hazard is not expected to occur during the lifetime of the medical device, likely to occur a few times, likely to occur frequently, etc) and the severity of that harm (death or loss of function or structure, reversible or minor injury, no injury or slight injury, etc). The scope of risk management is limited to the potential for harm to the patient, the care provider, and the treatment environment. More information on risk methodology can be found in the ISO 14971 standard, as referenced in Table 3.
      Effective software risk management covers different aspects. First, the risk management team, ideally involving developers but also users and clinicians, must identify and evaluate all possible risks associated with the device (including use errors). Second, manufacturers must take appropriate risk control measures to eliminate or minimize the risks. Finally, developers must demonstrate that the means taken to minimize the risks work as intended. This approach ensures that all technical solutions for safety are implemented in the software and that all information for safe use is consistent before the software is made available for clinical practice, while being compliant to the main international standards ISO 13485 on quality systems and ISO 14971 on risk management. Ultimately, the product’s overall residual risk must be acceptable, in relation to the state-of-the art, when used as intended by the manufacturer.
      Conformity of the developed MDSW to the General Safety and Performance Requirements must be proven by data. Evidence supporting safety and performance of the MDSW product is generated, in the first place, through verification and validation (V&V) activities as a part of the IEC 62304 software development process. The manufacturer provides first evidence of the ability of the device to generate the intended output accurately, reliably and precisely, from the input data. These activities rely on simulated or real-life training and validation data sets that act as inputs for the software/AI. Additionally, the product’s user requirements, specified in the intended use, must be validated by having the user use the MDSW in a simulated or real-life environment. The design verification and validation in the context of AI algorithms requires manufacturers to pay special attention to implementing appropriate controls over the training and test data such that bias in the dataset is avoided and that the accuracy of the model’s output is carefully validated. Note that for AI, the principles of human factors engineering may be applicable to traceability and explainability of the model, to allow a human expert to retrace the model’s decisions and use their judgment. Therefore, a user study to verify the extent to which the model can be interpreted by humans may become a necessary part of the validation [

      Johan Ordish, Hannah Murfet, Alison Hall. Algorithms as medical devices PHG Foundation (2019).

      ].
      The last part of the design and development process is the clinical evidence generation. Regardless of their classification, all medical devices require clinical evaluation. A clinical evaluation is a process conducted to demonstrate safety and performance as well as the overall positive benefit-risk ratio for a medical device, through critical evaluation of clinical data generated from the use of the device. Clinical data can be generated through clinical investigations of the concerned device or can be retrieved from the scientific literature or post-market surveillance, in particular through post-market clinical follow-up ( described later in this text). Data for comparable devices can be used only if equivalence to the subject device is demonstrated according to the strict criteria of the EU MDR, that require detailed comparison of clinical, technical, and biological (as applicable) characteristics and sufficient level of access to the technical documentation of the equivalent device.
      It is expected that the clinical data generated from the device is of sufficient amount (e.g., covers all indications and claims) and of sufficient quality (e.g. actuality of the dataset and appropriate statistical methods) to allow a comprehensive assessment of whether the device is safe and achieves the intended clinical benefit(s). This is also referred to as sufficient clinical evidence.
      The typical steps of a clinical evaluation process are shown in Fig. 2.
      Figure thumbnail gr2
      Fig. 2Clinical Evaluation of a medical device
      [

      European Commission. Guidelines on medical devices – clinical evaluation: a guide for manufacturers and notified bodies under directives 93/42/EEC and 90/385/EEC. MEDDEV 2.7/1 revision 4. http://ec.europa.eu/DocsRoom/documents/17522/attachments/1/translations/en/renditions/native. Accessed November 29, 2020.

      ]
      . CEP – Clinical Evaluation Plan, CER – Clinical Evaluation Report, PMCF – post-market clinical follow up, PMS – post-market surveillance.
      The first step is to define the evaluation scope and strategy in alignment with the medical intended purpose. This is captured in the clinical evaluation plan (CEP), which determines the type and amount of data required to support clinical claims, evaluate residual risks and demonstrate intended benefits. Manufacturers must justify the level of clinical evidence they consider sufficient, considering the state-of-the art. The latter is understood as current knowledge in the corresponding medical field, such as applicable standards and guidance documents, information related to the medical condition managed with the device, benchmark devices, and medical alternatives available to the target population. Description of state of the art, defined during a comprehensive and objective literature review, provides a framework to assess the device’s safety and performance.
      When identifying the pertinent data (i.e., data informing sufficient clinical evidence supporting intended clinical purpose and claims) in the clinical evaluation planning phase, a software’s manufacturer must consider three aspects crucial for its safe and effective use: valid clinical association, technical performance, and clinical performance (Fig. 3).
      Figure thumbnail gr3
      Fig. 3Components of clinical evidence for MDSW
      [

      MDCG 2020-1 Guidance on Clinical Evaluation (MDR) / Performance Evaluation (IVDR) of Medical Device Software. 2020.

      ]
      .
      The decision about whether or not a clinical investigation is required as a part of validation of clinical performance, is determined by the risks the device pose to patients and users, the clinical performance intended, and the claims of the manufacturer and the availability of existing clinical data, with the notable exception of class III (and implantable devices) where a clinical investigation is mandatory. Special attention should therefore be given to novel technologies with limited data on how the clinical care and patient outcomes are affected. Two main considerations for designing and conducting clinical investigations are protection of the rights, safety, dignity and well-being of the participating subjects and generation of clinical data that is scientifically valid, reliable & robust. These objectives are met by complying with the most recent version of the Declaration of Helsinki [
      • World Medical Association
      World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects.
      ] and ISO 14155 standard on good clinical practice for clinical studies with medical devices. Clinical investigations are subject to scientific & ethical review. The ethical review is performed by an institutional ethics committee in accordance with national law.
      During data appraisal, the evaluators are looking to make sure they have statistically significant data sets, where proper statistical methods and adequate controls were used, containing data that was generated in compliance with the relevant standards and local regulations (i.e. data of sufficient amount and quality). During the analysis stage, the evaluators conduct a comprehensive assessment to determine if the identified data meets the clinical safety and performance requirements relevant for the device. The results of this assessment, as well as all the inputs, and each clinical evaluation stage is captured in the Clinical Evaluation Report (CER), a mandatory part of the technical documentation for every medical device destined for the European market.

      3.3 Quality Management System (QMS) implementation

      Aside from a compliant product, manufacturers of medical devices, including software medical devices, need an EU MDR compliant quality management system. Such a system implemented in their organisation enables manufacturers to consistently deliver conforming products with the required quality, safety, and performance. For MDSW, this entails good software development and lifecycle practices (Fig. 4). The international standard ISO 13485 is an effective solution to meet the comprehensive requirements for a quality management system for manufacturing medical devices. It provides a practical foundation for manufacturers to address regulations and responsibilities as well as demonstrating a culture of quality and excellence for the development of medical devices.
      Figure thumbnail gr4
      Fig. 4Plan Do Check Act cycle representing ISO 13485 quality management for medical device software.
      This quality management system does not only document the core product realisation process-software development, which links with risk management and clinical evaluation, and resulting in the technical documentation for product approval- but also provides for supporting processes. Indeed, management needs to foresee (internal and external) human resources and infrastructure to complete the core product realisation process, and also install processes for monitoring and measuring of outputs of processes, subsequent analysis of those data and taking actions based on those data (for example, product improvement or corrective actions). One of those monitoring systems is post-market surveillance (PMS), described in a later section.

      3.4 Regulatory submission

      The data and evidence generated in the design and development phase, supplemented with the device description, labels and instructions for use constitutes the technical documentation. Additionally, as part of the technical documentation, the manufacturer must develop a comprehensive PMS plan to collect and evaluate data on the device from routine clinical use. This entails both reactive and proactive data sources and processes to monitor complaints and product risks. Technical documentation is finalised by writing the Declaration of Conformity where the manufacturer confirms that the product complies to the EU MDR.
      As soon as a manufacturer has implemented the Quality Management System and finalized the technical documentation, the notified body can start the conformity assessment to evaluate whether the device and the manufacturer’s quality management system meet the requirements of the regulation. The notified body accomplishes this by on-site audits and unannounced inspections at the manufacturer’s development/ production sites and by assessing the technical documentation on a sampling basis.
      Successful conformity assessment leads to issuing a CE certificate. The manufacturer completes the Declaration of Conformity with the CE certificate number and applies the CE marking with the notified body number on the product. However, the role of the notified body does not stop after the device is placed on the market. During the entire lifetime of the device, the notified body will perform quality system audits, assess how the manufacturer performs post-market surveillance, and evaluate how the findings feed into the clinical evaluation report. These audits affect whether the manufacturer’s device can retain the CE-marking.
      Before placing the device on the market, a registration of the manufacturer in the European database EUDAMED [

      EUDAMED. European Commission 2020. https://ec.europa.eu/health/md_eudamed/overview_en (accessed January 29, 2021).

      ] is required. Subsequently, the device must be registered, accompanied by a unique device identifier, with the goal of improving transparency and coordination of information regarding medical devices available on the EU market, both for market surveillance purposes by competent authorities and for general public information.

      3.5 Post-Market Surveillance (PMS)

      Although premarket product and QMS compliance receive a lot of attention from manufacturers who want to establish themselves in their target markets, successful commercialization also requires viable PMS efforts. Post-market surveillance aims to continuously verify the benefits of a medical device and to identify previously unknown risks by observing and analysing daily practical usage. Maintaining a PMS system is necessary not only to address the regulatory requirements but also to improve risk management and potentially improve the quality of the device. The EU MDR requires the PMS to be (pro)active – thus actively seeking to confirm the product’s real-world safety and performance. It also requires to be planned – based on the pre-defined plan as a part of the QMS; integration– effectively linked to other areas of the QMS; and lust be risk-based, proportionate to the risk class of the device.
      At a minimum, the manufacturer must keep a record of all customer complaints they receive. They must establish and document procedures for receiving, reviewing and evaluating customer complaints, and triggering a vigilance procedure, if needed. Incidents must be reported to competent authorities via the EUDAMED database, including the related corrective action. An incident has a very broad meaning. It does not necessarily mean that a person was seriously injured, both real and potential incidents need to be reported. It is sufficient that a serious injury could have happened or could happen if the event occurs again, with the same or an equivalent device. Examples of incidents that could be reportable include software that crashes, dosages that are too high, (for example because of double exposure or a wrong calibration), data being lost or corrupted, calculations or other functions that failed or an instruction that was omitted from the user manual.
      Due to the innovative character of AI tools, manufacturers often have a limited view of their adoptability, their clinical utility or the generalizability of their performance, which could bias the promised benefits. This limited knowledge may result in under or over representation of risks in the pre-market assessment of the device design and its interaction with the user. Post-market clinical follow-up (PMCF) may be required, based on the outcome of the clinical evaluation, to ensure adequate characterization of the real-world clinical use of the device. By nature, MDSW gathers many real-world performance data, which can be analysed and used for multiple purposes, including, but not limited, to timely detection and correction of MDSW malfunctions, detection of systematic misuse, understanding user interactions, monitoring clinical performance, and improving effectiveness.
      When a manufacturer desires to change a CE-certified medical device (for example, the intended use, the product claims, or the design) or an approved QMS, they are required to submit prior approval plans for changes to the notified body. The notified body will assess the proposed changes and decide whether they require a new conformity assessment or whether they could be addressed by means of a supplement to the EU technical documentation assessment certificate. In the latter case, the notified body shall assess the changes, notify the manufacturer of its decision and, where the changes are approved, provide it with a supplement to the EU technical documentation assessment certificate.

      3.6 Medical Device Software (MDSW) development for in-house use

      Health institutions may develop and use MDSW in their clinical practice if they can prove that the target patient group's specific needs cannot be met by an equivalent device or if the level of performance of an equivalent device available on the market is not appropriate. As health institutions do not intend to -and are not allowed to- commercialize their MDSW (including transfer of the MDSW to another legal entity), developing a commercialization strategy is not required.
      However, just like commercial manufacturers, health institutes must identify the applicable GSPRs and standards to follow during the product’s design and development. They should also maintain documentation of all evidence providing an understanding of the manufacturing facility, the manufacturing process, and the design, development and performance data of the device, including the intended purpose. The institution’s development, maintenance, and use of the MDSW must occur under an appropriate quality management system, although it does not require certification by a third party.
      A health institution as manufacturer does not need to obtain explicit premarket regulatory approval and thus does not need to provide a regulatory file to a notified body, but it does require a notification to the competent authority that can inspect the health institution’s activities and can restrict the manufacture and use of any particular type of device. Finally, a health institution needs to draw up a declaration which it has to make publicly available, containing its name and address and the identified in-house manufactured device, stating that the device meets the general safety and performance requirements of the EU MDR (or, where applicable, information on which requirements are not fully met with a reasoned justification therefor).
      After bringing the MDSW into clinical practice, the health institution should review experience gained from clinical use and take all necessary corrective actions.

      4. Discussion and conclusions

      MPEs should play a key role in any forum where surveillance of AI technology is currently being developed. With expertise positioned between physics, informatics and medicine, MPEs should become important stakeholders in the processes of developing and assessing innovative medical devices, including AI software applications for diagnostic imaging and radiation therapy.
      The role of the MPE is to contribute to patient safety and secure the quality of care, as established in the 2013/59/Euratom directive [

      Council Directive 2013/59/Euratom of 5 December 2013 laying down basic safety standards for protection against the dangers arising from exposure to ionising radiation, and repealing Directives 89/618/Euratom, 90/641/Euratom, 96/29/Euratom, 97/43/Euratom and 2003/122/Euratom n.d. https://eur-lex.europa.eu/eli/dir/2013/59/oj (accessed November 30, 2020).

      ]. This applies to radiation protection and dosimetry topics, but also extends to software devices that have an impact on the diagnosis or treatment of the patient, as stated in the EFOMP statement policy 16 [
      • Caruana C.J.
      • Tsapaki V.
      • Damilakis J.
      • Brambilla M.
      • Martín G.M.
      • Dimov A.
      • et al.
      EFOMP policy statement 16: the role and competences of medical physicists and medical physics experts under 2013/59/EURATOM.
      ] and in national transpositions of the Euratom directive [

      Federale Overheidsdienst Binnenlandse Zaken en Federaal Agentschap voor Nucleaire Controle. Koninklijk besluit betreffende de bescherming tegen ioniserende stralingen tijdens diergeneeskundige blootstellingen 2020/2000245. BELGISCH STAATSBLAD — 20.02.2020.

      ], where the MPE is given a role of innovator, and (re-)searcher to optimize the processes. This means that, just as with for x-ray devices, an AI software medical device with an impact on patient outcome, workflow or technology optimization, should also follow a process of procurement, commissioning and acceptance before introduction in clinical practice.
      The MPE should be involved in writing bid requests to ensure that the appropriate technical data are supplied by the vendors of the AI software, in a format permitting a reasonable comparison among the products's offer specifications. This would allow the radiologists, radiation oncologists and nuclear medicine physicians to make an informed decision about which software best matches given clinical needs. Once the purchase decision is made, the physicist should be involved in the commissioning of the AI software tool, by making a thorough evaluation of the system performance to be certain that the system meets the specifications of the manufacturer. Finally, the MPE can use acceptance test data to define baseline values required for future quality assurance performance testing.
      In case the AI tool is an in-house developed tool, the MPE will be using training, test- or validation data sets depending on the stages of the creation of the model.
      While up-to-date there are very few guidelines on how to commission or accept an AI MDSW for imaging or therapy purpose [
      • Mahadevaiah G.
      • Rv P.
      • Bermejo I.
      • Jaffray D.
      • Dekker A.
      • Wee L.
      Artificial intelligence-based clinical decision support in modern medical physics: selection, acceptance, commissioning, and quality assurance.
      ,

      Overview of artificial intelligence-based applications in radiotherapy: recommendations for implementation and quality assurance. Radiother Oncol. 2020 Dec;153:55–66. 10.1016/j.radonc.2020.09.008.

      ], it is of paramount importance that the MPEs become familiar with the pillars of the EU MDR, such that they can understand the full regulatory roadmap, from intended use, to risk classification, clinical evidence generation and post-market surveillance of such tools. It is by knowing how the AI application was developed, for which intended use and how clinical evidence for its performance was assessed that the MPE can understand how to purchase, commission and introduce the tool in clinical practice.
      One difficulty of this approach is that the AI applications currently on the market typically have a very narrow intended use. Even two applications aiming at similar clinical needs might be CE marked using different performance criteria, making it difficult to compare similar product (if available at all) using standard methods, such those used for x-ray equipment. For this reason and especially for quality assurance (QA) purposes over time, it is important to understand the path that led to a product’s regulatory approval and the evidence that was used to support its performance, including data used for training and performance metrics.
      The QA assurance over time might also be impacted by the type of software in terms of temporal changes., For Example some AI software are ‘locked’, so they provide the same result each time the same input is applied. However, other AI applications might change their performance during runtime, and in various ways: via a change by user to select an appropriate working point, e.g. to balance sensitivity vs. specificity of an algorithm; or through a discrete change through learning (an AI-based device that learns in the field whereby the updated model is activated via explicit manufacturer or user interaction); or a continuous change through learning: an AI-based device that learns in the field whereby the model is updated without explicit manufacturer or user interaction. These are key points for the MPE when setting in place a QA program for such tools and this information can for example be captured by understanding the clinical evidence generated for the CE marking. Moreover, if a collaboration with the hospital using the application is already set in place, the output of QA test could be used by the vendor for the PMS file.
      It is also important to note that, if a natural or legal person (1) changes the intended purpose or (2) modifies a device already placed on the market or put into service in such a way that compliance with the applicable requirements is affected, then that person must assume the obligations incumbent on manufacturers. In other words, when introducing a commercially available AI tool in a clinical workflow, it is not allowed to alter its intended use according to the needs of the clinic, and so this assessment should also be part of the QA of the system.
      Lastly, a thorough understanding of the regulatory requirements for AI software developed in-house, allows the MPE - as part of a multidisciplinary team- to translate the research into clinical practice.
      In conclusion, while some may advocate that AI will reduce the need for clinical medical physicists [
      • Tang X.
      • Wang B.
      • Rong Y.
      Artificial intelligence will reduce the need for clinical medical physicists.
      ], we think that MPEs, like other healthcare professionals, will be further empowered and some of our tasks will have to change or even expand. Up-skilling via dedicated training and practice will be key to paving the way to MPEs future roles and involvement in AI deployment in healthcare, including the regulatory aspects of these tools.

      References

        • Nensa F.
        • Demircioglu A.
        • Rischpler C.
        Artificial intelligence in nuclear medicine.
        J Nucl Med. 2019; 60: 29S-37Shttps://doi.org/10.2967/jnumed.118.220590
        • Jha S.
        • Cook T.
        Artificial intelligence in radiology-the state of the future.
        Acad Radiol. 2020; 27: 1-2https://doi.org/10.1016/j.acra.2019.11.003
        • Thompson R.F.
        • Valdes G.
        • Fuller C.D.
        • Carpenter C.M.
        • Morin O.
        • Aneja S.
        • et al.
        Artificial intelligence in radiation oncology: a specialty-wide disruptive transformation?.
        Radiother Oncol. 2018; 129: 421-426https://doi.org/10.1016/j.radonc.2018.05.030
        • Sahiner B.
        • Pezeshk A.
        • Hadjiiski L.M.
        • Wang X.
        • Drukker K.
        • Cha K.H.
        • et al.
        Deep learning in medical imaging and radiation therapy.
        Med Phys. 2019; 46: e1-e36https://doi.org/10.1002/mp.13264
        • Liu P.
        • Wang M.
        • Wang Y.
        • Yu M.
        • Wang Y.
        • Liu Z.
        • et al.
        Impact of deep learning-based optimization algorithm on image quality of low-dose coronary CT angiography with noise reduction: a prospective study.
        Acad Radiol. 2020; 27: 1241-1248https://doi.org/10.1016/j.acra.2019.11.010
        • McCollough C.H.
        • Leng S.
        Use of artificial intelligence in computed tomography dose optimisation.
        Ann ICRP. 2020; 014664532094082https://doi.org/10.1177/0146645320940827
        • McKinney S.M.
        • Sieniek M.
        • Godbole V.
        • Godwin J.
        • Antropova N.
        • Ashrafian H.
        • et al.
        International evaluation of an AI system for breast cancer screening.
        Nature. 2020; 577: 89-94https://doi.org/10.1038/s41586-019-1799-6
        • Geras K.J.
        • Mann R.M.
        • Moy L.
        Artificial intelligence for mammography and digital breast tomosynthesis: current concepts and future perspectives.
        Radiology. 2019; 293: 246-259https://doi.org/10.1148/radiol.2019182627
        • Summers R.M.
        • Beaulieu C.F.
        • Pusanik L.M.
        • Malley J.D.
        • Jeffrey R.B.
        • Glazer D.I.
        • et al.
        Automated polyp detector for CT colonography: feasibility study.
        Radiology. 2000; 216: 284-290https://doi.org/10.1148/radiology.216.1.r00jl43284
      1. Sohee Park, Sang Min Lee, Kyung Hee Lee, Kyu-Hwan Jung, Woong Bae, Jooae Choe Joon Beom Seo. Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings. Eur Radiol. 2020 Mar;30(3):1359–68. 10.1007/s00330-019-06532-x. Epub 2019 Nov 20.

        • Sim Y.
        • Chung M.J.
        • Kotter E.
        • Yune S.
        • Kim M.
        • Do S.
        • et al.
        Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs.
        Radiology. 2020; 294: 199-209https://doi.org/10.1148/radiol.2019182465
        • Tandel G.S.
        • Balestrieri A.
        • Jujaray T.
        • Khanna N.N.
        • Saba L.
        • Suri J.S.
        Multiclass magnetic resonance imaging brain tumor classification using artificial intelligence paradigm.
        Comput Biol Med. 2020; 122: 103804https://doi.org/10.1016/j.compbiomed.2020.103804
        • Hu W.
        • Yang H.
        • Xu H.
        • Mao Y.
        Radiomics based on artificial intelligence in liver diseases: where we are?.
        Gastroenterol Rep (Oxf). 2020; 8: 90-97https://doi.org/10.1093/gastro/goaa011
      2. Calculating the target exposure index using a deep convolutional neural network and a rule base. Phys Med. 2020 Mar;71:108–14. 10.1016/j.ejmp.2020.02.012. Epub 2020 Feb 27.

        • Pagnozzi A.M.
        • Fripp J.
        • Rose S.E.
        Quantifying deep grey matter atrophy using automated segmentation approaches: a systematic review of structural MRI studies.
        Neuroimage. 2019; 201: 116018https://doi.org/10.1016/j.neuroimage.2019.116018
      3. Simple low-cost approaches to semantic segmentation in radiation therapy planning for prostate cancer using deep learning with non-contrast planning CT images. Phys Med. 2020 Oct;78:93–100. 10.1016/j.ejmp.2020.09.004. Epub 2020 Sep 17.

        • Lakhani P.
        • Prater A.B.
        • Hutson R.K.
        • Andriole K.P.
        • Dreyer K.J.
        • Morey J.
        • et al.
        Machine learning in radiology: applications beyond image interpretation.
        J Am Coll Radiol. 2018; 15: 350-359https://doi.org/10.1016/j.jacr.2017.09.044
      4. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance.). vol. 117. 2017.

      5. Council Directive 93/42/EEC of 14 June 1993 concerning medical devices. vol. OJ L. 1993.

        • Sensakovic W.F.
        • Mahesh M.
        Role of the medical physicist in the health care artificial intelligence revolution.
        J Am College Radiol. 2019; 16: 393-394https://doi.org/10.1016/j.jacr.2018.09.022
      6. Council Directive 2013/59/Euratom of 5 December 2013 laying down basic safety standards for protection against the dangers arising from exposure to ionising radiation, and repealing Directives 89/618/Euratom, 90/641/Euratom, 96/29/Euratom, 97/43/Euratom and 2003/122/Euratom n.d. https://eur-lex.europa.eu/eli/dir/2013/59/oj (accessed November 30, 2020).

      7. EC. RP174 European Guidelines on the Medical Physics Expert. 2014. http://op.europa.eu/en/publication-detail/-/publication/b82ed768-4c50-4c9a-a789-98a3b0df5391 (accessed January 29, 2021).

      8. mdcg_2019_11_guidance_qualification_classification_software (1).pdf n.d.

      9. Regulation (EU) 2016/ 679 of the European Parliament and of the Council - of 27 April 2016 - on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/ 46/ EC (General Data Protection Regulation). n.d.:88.

        • Vanegas E.
        • Igual R.
        • Plaza I.
        Sensing systems for respiration monitoring: a technical systematic review.
        Sensors (Basel). 2020; 20https://doi.org/10.3390/s20185446
        • Li K.H.C.
        • White F.A.
        • Tipoe T.
        • Liu T.
        • Wong M.C.
        • Jesuthasan A.
        • et al.
        The current state of mobile phone apps for monitoring heart rate, heart rate variability, and atrial fibrillation: narrative review.
        JMIR Mhealth Uhealth. 2019; 7: e11606https://doi.org/10.2196/11606
        • Ding X.-R.
        • Clifton D.
        • Ji N.
        • Lovell N.H.
        • Bonato P.
        • Chen W.
        • et al.
        Wearable sensing and telehealth technology with potential applications in the coronavirus pandemic.
        IEEE Rev Biomed Eng. 2020; https://doi.org/10.1109/RBME.2020.2992838
      10. “Software as a Medical Device”: possible framework for risk categorization and corresponding considerations 2014:30.

      11. Johan Ordish, Hannah Murfet, Alison Hall. Algorithms as medical devices PHG Foundation (2019).

      12. European Commission. Guidelines on medical devices – clinical evaluation: a guide for manufacturers and notified bodies under directives 93/42/EEC and 90/385/EEC. MEDDEV 2.7/1 revision 4. http://ec.europa.eu/DocsRoom/documents/17522/attachments/1/translations/en/renditions/native. Accessed November 29, 2020.

      13. MDCG 2020-1 Guidance on Clinical Evaluation (MDR) / Performance Evaluation (IVDR) of Medical Device Software. 2020.

        • World Medical Association
        World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects.
        JAMA. 2013; 310: 2191-2194https://doi.org/10.1001/jama.2013.281053
      14. EUDAMED. European Commission 2020. https://ec.europa.eu/health/md_eudamed/overview_en (accessed January 29, 2021).

        • Caruana C.J.
        • Tsapaki V.
        • Damilakis J.
        • Brambilla M.
        • Martín G.M.
        • Dimov A.
        • et al.
        EFOMP policy statement 16: the role and competences of medical physicists and medical physics experts under 2013/59/EURATOM.
        Phys Med. 2018; 48: 162-168https://doi.org/10.1016/j.ejmp.2018.03.001
      15. Federale Overheidsdienst Binnenlandse Zaken en Federaal Agentschap voor Nucleaire Controle. Koninklijk besluit betreffende de bescherming tegen ioniserende stralingen tijdens diergeneeskundige blootstellingen 2020/2000245. BELGISCH STAATSBLAD — 20.02.2020.

        • Mahadevaiah G.
        • Rv P.
        • Bermejo I.
        • Jaffray D.
        • Dekker A.
        • Wee L.
        Artificial intelligence-based clinical decision support in modern medical physics: selection, acceptance, commissioning, and quality assurance.
        Med Phys. 2020; 47: e228-e235https://doi.org/10.1002/mp.13562
      16. Overview of artificial intelligence-based applications in radiotherapy: recommendations for implementation and quality assurance. Radiother Oncol. 2020 Dec;153:55–66. 10.1016/j.radonc.2020.09.008.

        • Tang X.
        • Wang B.
        • Rong Y.
        Artificial intelligence will reduce the need for clinical medical physicists.
        J Appl Clin Med Phys. 2018; 19: 6-9https://doi.org/10.1002/acm2.12244