Advertisement

Basic of machine learning and deep learning in imaging for medical physicists

Published:April 04, 2021DOI:https://doi.org/10.1016/j.ejmp.2021.03.026

      Highlights

      • An overview of AI tools applied to imaging for Healthcare is presented.
      • Code and available database for developing AI strategies are indicated.
      • The limitation and opportunity of AI application is discussed.
      • Explainable AI represents the new frontiers for penetrating the “black box” of ML algorithms.

      Abstract

      The manuscript aims at providing an overview of the published algorithms/automation tool for artificial intelligence applied to imaging for Healthcare. A PubMed search was performed using the query string to identify the proposed approaches (algorithms/automation tools) for artificial intelligence (machine and deep learning) in a 5-year period. The distribution of manuscript in the various disciplines and the investigated image types according to the AI approaches are presented. The limitation and opportunity of AI application in the clinical practice or in the next future research is discussed.

      Keywords

      Introduction

      The trend in medical imaging is an ever-increasing amount of information that can be viewed as a derived demand in each discipline. Improvements in computer technology and performance have made information processing faster, its speed has churned more information for physician to process.
      Artificial intelligence (AI) is a branch of computer sciences [

      Ranschaert E, Morozov S, Algra P. Artificial Intelligence in Medical Imaging Opportunities, Applications and Risks: Opportunities, Applications and Risks; 2019.

      ]. Its objective is the development of machines whose cognitive functions related to mimic the perception, learning, problem-solving and decision-making exceed that of humans. AI is the entire universe of computer technology that exhibits anything remotely resembling human intelligence.
      Machine learning (ML) is a branch of AI [
      • Lee J.G.
      • Jun S.
      • Cho Y.W.
      • Lee H.
      • Kim G.B.
      • Seo J.B.
      • et al.
      Deep learning in medical imaging: general overview.
      ] in which, based on the training dataset that are first provided, the computer develops its own logic for answering future questions. Given the large volume of data generated in many domains of science, AI appears uniquely advantageous for analysis of OMIC studies [
      • Shi L.
      • Kusko R.
      • Wolfinger R.D.
      • Haibe-Kains B.
      • Fischer M.
      • Sansone S.A.
      • et al.
      The international MAQC Society launches to enhance reproducibility of high-throughput technologies.
      ]. The key concept of machine learning is to produce accurate predictions on new unseen data after being trained on a finite learning dataset. To concretely define what learning means, Mitchell et al. proposed to specify three parameters, namely T, P, and E [
      • Mitchell T.M.
      Machine learning.
      ]: “a computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance on tasks in T, as measured by P, improves with experience E.” [
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ].
      Advances in ML techniques may facilitate processing of large amounts of medical image data. Notwithstanding their utility, ML methods are known to have limitations [
      • Sedik A.
      • Iliyasu A.M.
      • Abd El-Rahiem B.
      • Abdel Samea M.E.
      • Abdel-Raheem A.
      • Hammad M.
      • et al.
      Deploying machine and deep learning models for efficient data-augmented detection of COVID-19 infections.
      ] related to: manual extraction and selection of features, this is a fundamental task in order to find a group of significant variables to predict and correlate with outcome; Poor performance when dealing with imbalanced datasets; Over-fitting generating a model that corresponds exactly to the input training set of data and may therefore fail to fit new data, when this occurs, the algorithm will be accurate (low error rate between predicted and actual results) on the training data set, but highly inaccurate on the testing data set; Complexity and time consumption, it is important to remind that the AI model can aid the human decision process for example in routine medical application only if their results are robust and obtainable in a reasonable computational time.
      Deep learning (DL) is the newest class of ML and has been found to be advantageous to other forms ofML [
      • Min S.
      • Lee B.
      • Yoon S.
      Deep learning in bioinformatics.
      ]. DL employs multiple layers of neural networks, leading to expanded ‘neuronal’ complexity, to significantly enhance computational power. DL has been recently applied to bioinformatics.
      Furthermore, deep learning has ability to find out irrelevant and particular minute variations, which allows these methods to reach higher accuracy than other machine learning methods [
      • Schmidhuber J.
      Deep learning in neural networks: an overview.
      ]. Among deep-learning methods, Convolutional Neural Networks (CNNs) show superior image recognition ability [
      • Motozawa N.
      • An G.
      • Takagi S.
      • Kitahata S.
      • Mandai M.
      • Hirami Y.
      • et al.
      Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes.
      ].
      Deep neural networks (DNNs) are forms of machine learning methods. In machine learning, it is often necessary to reduce the complexity of the input data and make relevant patterns more visible forthe learning algorithms to function. Indeed, their performance greatly depends on how accurately these features have been identified and extracted. Given that this feature engineering process is based on domain knowledge and is specific to the data type, it is difficult and expensive in terms of time and expertise to apply. In contrast, DNNs independently learn a hierarchical representation of the input data adapted to the task at hand; this eliminates the task of developing new features extractors for every problem. The main drawback is that DNNs require a large amount of input data to be effective. The idea of a computer program that could find itself representing a model from a dataset is not new. Perceptron isone of the first approaches to conceptualize data directly from the environment
      As discussed by Meyer et al the DNN’s performance greatly depends on how accurately the features have been identified and extracted. Using DNNs the task of developing new features extractors for specific problem is not necessary because the algorithm independently learn a hierarchical representation of the input data. To perform these task, the DNN based algorithms require a large number of input data [
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ].
      The predictive power on artificial neural networks (ANNs) depends on the number of multiple layers of information allowing the analysis of datasets with more complex patterns. These algorithms are enabling pivotal advances in text/speech and image recognition.
      ML and DL algorithms use different learning approaches, as discussed by Meyer et al.[
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ] the most important methods are supervised, unsupervised, reinforcement, and transfer learning. In supervised learning, the algorithm uses training inputs and their corresponding outputs in order to learn a rule that connects inputs to outputs. In unsupervised learning, no referenced outputs are used by the algorithm to find a structure in the inputs. Reinforcement learning is based on communication and exploration; a feedback is used to adjust the learning process. In cases in which the available input data are insufficient to train a network (i.e. medical applications), transfer learning can be used to adapt a pre-trained network to perform a new task; the fine-tune of the network weights can be performed using a small amount of coherent labeled data [
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ].
      Supervised learning techniques are undoubtedly the most widely used methods in ML and those with the best results. These procedures rely on a dataset from which the response variable to be predicted (eg. diagnosis, parameter, segmentation) by two or more classes using a series of variables for prediction and classification algorithms or continuous values for regressive algorithms.
      Unfortunately, AI tools suffer from an inability to correctly detect and classify cases that they have not previously analyzed. Thus, the reliability and quality of the data source are essential for an algorithm to be realistic, correct, not biased, and applicable into realistic clinical contexts. The images are generally labeled with two or more conditions (e.g. healthy/sick) by two or more experts or through radiology/specialists reports.
      With the advancement of artificial intelligence in medical diagnostics, many studies have been devoted to compare whether AI or diagnostician which is expected to be better than diagnostician alone. Thanks to these approaches often several hundred measurable features which are beyond the search limits of the human retina and require computational power for both detection and analysis are available for each image. Thus, the specialists are overwhelmed not just with what they can see but with what is exceeding their cognitive bandwidth [
      • Lee C.S.
      • Nagy P.G.
      • Weaver S.J.
      • Newman-Toker D.E.
      Cognitive and system factors contributing to diagnostic errors in radiology.
      ].
      Another trend is the evolution from visual subjectivity to quantitative objectivity, which reduces the inter-observer variation with this subjective assessment, while improving the consistency of parameters which are objectively quantified.
      The impact of AI on the various disciplines depends on the balance between how fast it automates old tasks and how many new information it generates. In the short-term, AI may create more work for experts because it may increase information at a faster rate than it removes work by automating tasks.
      To evaluate the state of the art of ML and DL methods applied to medical imaging in Healthcare, a PubMed search has been conducted with the purpose of discussing the actually implemented methods, the investigated images, highlighting the efforts in multicentric initiative and the limitation of studies as strategy to reinforce the technology pillar.

      Methods

      Literature search strategy

      A PubMed search was performed using the query string to identify the proposed approaches (algorithms/automation tools) for artificial intelligence (machine and deep learning) applied to medical imaging for Healthcare. Query search included the following keywords/string:
      Search: (algorithm OR automation) AND “deep learning” AND “machine learning” AND “artificial intelligence” NOT dental NOT review Filters: from 2015/9/272020/9/28. The research was restricted to the last five years to include only the most recently published studies. The search was done on 28thSeptember 2020.

      Study selection

      Titles and abstracts were independently reviewed by three authors to decide study inclusion. Full articles were retrieved when the abstract was considered relevant and only papers or abstracts published in English were considered. Papers were considered eligible when 1) the main and/or secondary study aim included the application of algorithms/automation tools for artificial intelligence applied to medical imaging or Healthcare; 2) involved a number of patients of training/test set >30 or investigated at least one type of image modality with a number of images >50. The data were summarized in a database with the following issues: first author, journal, year, title, discipline/area, exclusion/issues, AI area, number of images, number of patients, training set size, test set size, validation set size, algorithm, main aim (classification, detection, pattern recognition, predict outcome, segmentations), image type, specific image type, code algorithm, secondary study aim (improve accuracy/knowledge/performances/efficiency, comparison, other), for the subsequent data analysis.

      Results

      Description of included studies and inclusion criteria

      Based on reported PubMed/Medline search, 169 paper and abstracts reporting or proposing algorithms/automation tool for artificial intelligence (machine and deep learning) applied to medical imaging for Healthcare were identified. The number of all identified papers per year increased as shown in Fig. 1 according to the inclusion/exclusion judgment. An exponential growth can be observed looking at the included papers trough years. The number of papers related to the 2020 was not complete due to the inclusion criteria limited to the 2020/9/28.
      Figure thumbnail gr1
      Fig. 1The number of identified papers per year according to inclusion (yes) and exclusion (no) judgment identified from 2015 to 09-27 to 2020-09-28.
      The results of the PubMed search were represented in a PRISMA flow-diagram of study selection and inclusion [
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      ] as reported in Fig. 2. Eighty-two papers of 169 records [
      • Sedik A.
      • Iliyasu A.M.
      • Abd El-Rahiem B.
      • Abdel Samea M.E.
      • Abdel-Raheem A.
      • Hammad M.
      • et al.
      Deploying machine and deep learning models for efficient data-augmented detection of COVID-19 infections.
      ,
      • Motozawa N.
      • An G.
      • Takagi S.
      • Kitahata S.
      • Mandai M.
      • Hirami Y.
      • et al.
      Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes.
      ,
      • Trivizakis E.
      • Ioannidis G.S.
      • Melissianos V.D.
      • Papadakis G.Z.
      • Tsatsakis A.
      • Spandidos D.A.
      • et al.
      A novel deep learning architecture outperforming 'off-the-shelf' transfer learning and feature-based methods in the automated assessment of mammographic breast density.
      ,
      • VoPham T.
      • Hart J.E.
      • Laden F.
      • Chiang Y.Y.
      Emerging trends in geospatial artificial intelligence (geoAI): potential applications for environmental epidemiology.
      ,
      • Hoseini F.
      • Shahbahrami A.
      • Bayat P.
      AdaptAhead optimization algorithm for learning deep CNN applied to MRI segmentation.
      ,
      • Li H.
      • Parikh N.A.
      • He L.
      A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes.
      ,
      • Kiani A.
      • Uyumazturk B.
      • Rajpurkar P.
      • Wang A.
      • Gao R.
      • Jones E.
      • et al.
      Impact of a deep learning assistant on the histopathologic classification of liver cancer.
      ,
      • Mobadersany P.
      • Yousefi S.
      • Amgad M.
      • Gutman D.A.
      • Barnholtz-Sloan J.S.
      • Velázquez Vega J.E.
      • et al.
      Predicting cancer outcomes from histology and genomics using convolutional networks.
      ,
      • Ovchinnikova K.
      • Kovalev V.
      • Stuart L.
      • Alexandrov T.
      OffsampleAI: artificial intelligence approach to recognize off-sample mass spectrometry images.
      ,
      • Borkowski A.A.
      • Viswanadhan N.A.
      • Thomas L.B.
      • Guzman R.D.
      • Deland L.A.
      • Mastorides S.M.
      Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis.
      ,
      • Fares A.
      • Zhong S.H.
      • Jiang J.
      EEG-based image classification via a region-level stacked bi-directional deep learning framework.
      ,
      • Huang Y.
      • Xu J.
      • Zhou Y.
      • Tong T.
      • Zhuang X.
      Diagnosis of Alzheimer's disease via multi-modality 3D convolutional neural network.
      ,
      • Jin D.
      • Zhou B.
      • Han Y.
      • Ren J.
      • Han T.
      • Liu B.
      • et al.
      Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer's disease.
      ,
      • Lawrimore J.
      • Doshi A.
      • Walker B.
      • Bloom K.
      AI-assisted forward modeling of biological structures.
      ,
      • Kusumoto D.
      • Lachmann M.
      • Kunihiro T.
      • Yuasa S.
      • Kishino Y.
      • Kimura M.
      • et al.
      Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells.
      ,
      • Fischer C.A.
      • Besora-Casals L.
      • Rolland S.G.
      • Haeussler S.
      • Singh K.
      • Duchen M.
      • et al.
      MitoSegNet: easy-to-use deep learning segmentation for analyzing mitochondrial morphology.
      ,
      • Zhang Y.
      • Hamada M.
      DeepM6ASeq: prediction and characterization of m6A-containing sequences using deep learning.
      ,
      • Yamada K.D.
      • Kinoshita K.
      De novo profile generation based on sequence context specificity with the long short-term memory network.
      ,
      • Ho Thanh Lam L.
      • Le N.H.
      • Van Tuan L.
      • Tran Ban H.
      • Nguyen Khanh Hung T.
      • Nguyen N.T.K.
      • et al.
      Machine learning model for identifying antioxidant proteins using features calculated from primary sequences.
      ,
      • Yao K.
      • Rochman N.D.
      • Sun S.X.
      CTRL - a label-free artificial intelligence method for dynamic measurement of single-cell volume.
      ,
      • Smolander J.
      • Stupnikov A.
      • Glazko G.
      • Dehmer M.
      • Emmert-Streib F.
      Comparing biological information contained in mRNA and non-coding RNAs for classification of lung cancer patients.
      ,
      • Kwon J.M.
      • Kim K.H.
      • Jeon K.H.
      • Kim H.M.
      • Kim M.J.
      • Lim S.M.
      • et al.
      Development and validation of deep-learning algorithm for electrocardiography-based heart failure identification.
      ,
      • Kwon J.M.
      • Kim K.H.
      • Jeon K.H.
      • Lee S.E.
      • Lee H.Y.
      • Cho H.J.
      • et al.
      Artificial intelligence algorithm for predicting mortality of patients with acute heart failure.
      ,
      • Kwon J.M.
      • Lee Y.
      • Lee Y.
      • Lee S.
      • Park J.
      An Algorithm based on deep learning for predicting in-hospital cardiac arrest.
      ,
      • Diller G.P.
      • Babu-Narayan S.
      • Li W.
      • Radojevic J.
      • Kempny A.
      • Uebing A.
      • et al.
      Utility of machine learning algorithms in assessing patients with a systemic right ventricle.
      ,
      • Lin C.S.
      • Lin C.
      • Fang W.H.
      • Hsu C.J.
      • Chen S.J.
      • Huang K.H.
      • et al.
      A Deep-Learning Algorithm (ECG12Net) for detecting hypokalemia and hyperkalemia by electrocardiography: algorithm development.
      ,
      • Attia Z.I.
      • Sugrue A.
      • Asirvatham S.J.
      • Ackerman M.J.
      • Kapa S.
      • Friedman P.A.
      • et al.
      Noninvasive assessment of dofetilide plasma concentration using a deep learning (neural network) analysis of the surface electrocardiogram: a proof of concept study.
      ,
      • Hart S.N.
      • Flotte W.
      • Norgan A.P.
      • Shah K.K.
      • Buchan Z.R.
      • Mounajjed T.
      • et al.
      Classification of melanocytic lesions in selected and whole-slide images via convolutional neural networks.
      ,
      • Hekler A.
      • Utikal J.S.
      • Enk A.H.
      • Hauschild A.
      • Weichenthal M.
      • Maron R.C.
      • et al.
      Superior skin cancer classification by the combination of human and artificial intelligence.
      ,
      • Walker B.N.
      • Rehg J.M.
      • Kalra A.
      • Winters R.M.
      • Drews P.
      • Dascalu J.
      • et al.
      Dermoscopy diagnosis of cancerous lesions utilizing dual deep learning algorithms via visual and audio (sonification) outputs: laboratory and prospective observational studies.
      ,
      • Maron R.C.
      • Utikal J.S.
      • Hekler A.
      • Hauschild A.
      • Sattler E.
      • Sondermann W.
      • et al.
      Artificial intelligence and its effect on dermatologists' accuracy in dermoscopic melanoma image classification: web-based survey study.
      ,
      • Han S.S.
      • Park G.H.
      • Lim W.
      • Kim M.S.
      • Na J.I.
      • Park I.
      • et al.
      Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
      ,
      • Bizzego A.
      • Bussola N.
      • Chierici M.
      • Maggio V.
      • Francescatto M.
      • Cima L.
      • et al.
      Evaluating reproducibility of AI algorithms in digital pathology with DAPPER.
      ,
      • Jones A.D.
      • Graff J.P.
      • Darrow M.
      • Borowsky A.
      • Olson K.A.
      • Gandour-Edwards R.
      • et al.
      Impact of pre-analytical variables on deep learning accuracy in histopathology.
      ,
      • Ehteshami Bejnordi B.
      • Veta M.
      • Johannes van Diest P.
      • van Ginneken B.
      • Karssemeijer N.
      • Litjens G.
      • et al.
      Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
      ,
      • Palatnik de Sousa I.
      • Maria Bernardes Rebuzzi Vellasco M.
      • Costa da Silva E.
      Local interpretable model-agnostic explanations for classification of lymph node metastases.
      ,
      • Vandenberghe M.E.
      • Scott M.L.
      • Scorer P.W.
      • Söderberg M.
      • Balcerzak D.
      • Barker C.
      Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer.
      ,
      • Khosravi P.
      • Kazemi E.
      • Zhan Q.
      • Malmsten J.E.
      • Toschi M.
      • Zisimopoulos P.
      • et al.
      Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization.
      ,
      • VerMilyea M.
      • Hall J.M.M.
      • Diakiw S.M.
      • Johnston A.
      • Nguyen T.
      • Perugini D.
      • et al.
      Development of an artificial intelligence-based assessment model for prediction of embryo viability using static images captured by optical light microscopy during IVF.
      ,
      • Takahashi H.
      • Tampo H.
      • Arai Y.
      • Inoue Y.
      • Kawashima H.
      Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy.
      ,
      • Heydon P.
      • Egan C.
      • Bolter L.
      • Chambers R.
      • Anderson J.
      • Aldington S.
      • et al.
      Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients.
      ,
      • Shah P.
      • Mishra D.K.
      • Shanmugam M.P.
      • Doshi B.
      • Jayaraj H.
      • Ramanjulu R.
      Validation of Deep Convolutional Neural Network-based algorithm for detection of diabetic retinopathy - Artificial intelligence versus clinician for screening.
      ,
      • Makino M.
      • Yoshimoto R.
      • Ono M.
      • Itoko T.
      • Katsuki T.
      • Koseki A.
      • et al.
      Artificial intelligence predicts the progression of diabetic kidney disease using big data machine learning.
      ,
      • Sánchez-DelaCruz E.
      • Weber R.
      • Biswal R.R.
      • Mejía J.
      • Hernández-Chan G.
      • Gómez-Pozos H.
      Gait biomarkers classification by combining assembled algorithms and deep learning: results of a local study.
      ,
      • Trivedi H.
      • Mesterhazy J.
      • Laguna B.
      • Vu T.
      • Sohn J.H.
      Automatic determination of the need for intravenous contrast in musculoskeletal MRI examinations Using IBM Watson's natural language processing algorithm.
      ,
      • Schaffter T.
      • Buist D.S.M.
      • Lee C.I.
      • Nikulin Y.
      • Ribli D.
      • Guan Y.
      • et al.
      Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.
      ,
      • Lee H.
      • Troschel F.M.
      • Tajmir S.
      • Fuchs G.
      • Mario J.
      • Fintelmann F.J.
      • et al.
      Pixel-level deep segmentation: artificial intelligence quantifies muscle on computed tomography for body morphometric analysis.
      ,
      • Pang S.
      • Ding T.
      • Qiao S.
      • Meng F.
      • Wang S.
      • Li P.
      • et al.
      A novel YOLOv3-arch model for identifying cholelithiasis and classifying gallstones on CT images.
      ,
      • Pasa F.
      • Golkov V.
      • Pfeiffer F.
      • Cremers D.
      • Pfeiffer D.
      Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization.
      ,
      • Cheng C.T.
      • Ho T.Y.
      • Lee T.Y.
      • Chang C.C.
      • Chou C.C.
      • Chen C.C.
      • et al.
      Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs.
      ,
      • Almog Y.A.
      • Rai A.
      • Zhang P.
      • Moulaison A.
      • Powell R.
      • Mishra A.
      • et al.
      Deep learning with electronic health records for short-term fracture risk identification: crystal bone algorithm development and validation.
      ,
      • Hu T.H.
      • Huo Z.
      • Liu T.A.
      • Wang F.
      • Wan L.
      • Wang M.W.
      • et al.
      Automated assessment for bone age of left wrist joint in uyghur teenagers by deep learning.
      ,
      • Lee H.
      • Tajmir S.
      • Lee J.
      • Zissen M.
      • Yeshiwas B.A.
      • Alkasab T.K.
      • et al.
      Fully automated deep learning system for bone age assessment.
      ,
      • Yune S.
      • Lee H.
      • Kim M.
      • Tajmir S.H.
      • Gee M.S.
      • Do S.
      Beyond human perception: sexual dimorphism in hand and wrist radiographs is discernible by a deep learning model.
      ,
      • Bangalore Yogananda C.G.
      • Shah B.R.
      • Vejdani-Jahromi M.
      • Nalawade S.S.
      • Murugesan G.K.
      • Yu F.F.
      • et al.
      A fully automated deep learning network for brain tumor segmentation.
      ,
      • Ardakani A.A.
      • Kanafi A.R.
      • Acharya U.R.
      • Khadem N.
      • Mohammadi A.
      Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: results of 10 convolutional neural networks.
      ,
      • Kuwayama S.
      • Ayatsuka Y.
      • Yanagisono D.
      • Uta T.
      • Usui H.
      • Kato A.
      • et al.
      Automated detection of macular diseases by optical coherence tomography and artificial intelligence machine learning of optical coherence tomography images.
      ,
      • Burlina P.M.
      • Joshi N.
      • Pekala M.
      • Pacheco K.D.
      • Freund D.E.
      • Bressler N.M.
      Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks.
      ,
      • Bajwa M.N.
      • Malik M.I.
      • Siddiqui S.A.
      • Dengel A.
      • Shafait F.
      • Neumeier W.
      • et al.
      Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning.
      ,
      • Christopher M.
      • Nakahara K.
      • Bowd C.
      • Proudfoot J.A.
      • Belghith A.
      • Goldbaum M.H.
      • et al.
      Effects of study population, labeling and training on glaucoma detection using deep learning algorithms.
      ,
      • Mariottoni E.B.
      • Datta S.
      • Dov D.
      • Jammal A.A.
      • Berchuck S.I.
      • Tavares I.M.
      • et al.
      Artificial intelligence mapping of structure to function in glaucoma.
      ,
      • Bahado-Singh R.O.
      • Vishweswaraiah S.
      • Aydas B.
      • Mishra N.K.
      • Guda C.
      • Radhakrishna U.
      Deep learning/artificial intelligence and blood-based DNA Epigenomic prediction of cerebral palsy.
      ,
      • Jin B.
      • Qu Y.
      • Zhang L.
      • Gao Z.
      Diagnosing Parkinson disease through facial expression recognition: video analysis.
      ,
      • Yang Y.
      • Ye Z.
      • Su Y.
      • Zhao Q.
      • Li X.
      • Ouyang D.
      Deep learning for in vitro prediction of pharmaceutical formulations.
      ,
      • Gao M.
      • Igata H.
      • Takeuchi A.
      • Sato K.
      • Ikegaya Y.
      Machine learning-based prediction of adverse drug effects: an example of seizure-inducing compounds.
      ,
      • Korotcov A.
      • Tkachenko V.
      • Russo D.P.
      • Ekins S.
      Comparison of deep learning with multiple machine learning methods and metrics using diverse drug discovery data sets.
      ,
      • Saez Y.
      • Baldominos A.
      • Isasi P.
      A comparison study of classifier algorithms for cross-person physical activity recognition.
      ,
      • Livingstone D.
      • Talai A.S.
      • Chau J.
      • Forkert N.D.
      Building an Otoscopic screening prototype tool using deep learning.
      ,
      • Khalifa N.E.M.
      • Taha M.H.N.
      • Manogaran G.
      • Loey M.
      A deep learning model and machine learning methods for the classification of potential coronavirus treatments on a single human cell.
      ,
      • Smail L.C.
      • Dhindsa K.
      • Braga L.H.
      • Becker S.
      • Sonnadara R.R.
      Using deep learning algorithms to grade hydronephrosis severity: toward a clinical adjunct.
      ,
      • Wang C.J.
      • Hamm C.A.
      • Savic L.J.
      • Ferrante M.
      • Schobert I.
      • Schlachter T.
      • et al.
      Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features.
      ,
      • Zimmer D.
      • Schneider K.
      • Sommer F.
      • Schroda M.
      • Mühlhaus T.
      Artificial intelligence understands peptide observability and assists with absolute protein quantification.
      ,
      • Weng S.
      • Xu X.
      • Li J.
      • Wong S.T.C.
      Combining deep learning and coherent anti-Stokes Raman scattering imaging for automated differential diagnosis of lung cancer.
      ,
      • Im H.
      • Pathania D.
      • McFarland P.J.
      • Sohani A.R.
      • Degani I.
      • Allen M.
      • et al.
      Design and clinical validation of a point-of-care device for the diagnosis of lymphoma via contrast-enhanced microholography and machine learning.
      ,
      • He H.
      • Yang X.
      • Xu Z.
      • Deng N.
      • Shang Y.
      • Liu G.
      • et al.
      Implementing artificial neural networks through bionic construction.
      ,
      • Bălan O.
      • Moise G.
      • Moldoveanu A.
      • Leordeanu M.
      • Moldoveanu F.
      Fear level classification based on emotional dimensions and machine learning techniques.
      ,
      • Spandorfer A.
      • Branch C.
      • Sharma P.
      • Sahbaee P.
      • Schoepf U.J.
      • Ravenel J.G.
      • et al.
      Deep learning to convert unstructured CT pulmonary angiography reports into structured reports.
      ,
      • Banerjee I.
      • Ling Y.
      • Chen M.C.
      • Hasan S.A.
      • Langlotz C.P.
      • Moradzadeh N.
      • et al.
      Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification.
      ,
      • Huang Z.
      • Liu D.
      • Chen X.
      • Yu P.
      • Wu J.
      • Song B.
      • et al.
      Retrospective imaging studies of gastric cancer: study protocol clinical trial (SPIRIT Compliant).
      ,
      • Waisman A.
      • La Greca A.
      • Möbbs A.M.
      • Scarafía M.A.
      • Santín Velazque N.L.
      • Neiman G.
      • et al.
      Deep learning neural networks highly predict very early onset of pluripotent stem cell differentiation.
      ,
      • Min X.
      • Yu B.
      • Wang F.
      Predictive modeling of the hospital readmission risk from patients' claims data using machine learning: a case study on COPD.
      ,
      • Jakobs M.A.
      • Dimitracopoulos A.
      • KymoButler F.K.
      a deep learning software for automated kymograph analysis.
      ,
      • Wachinger C.
      • Reuter M.
      • Klein T.
      DeepNAT: deep convolutional neural network for segmenting neuroanatomy.
      ,
      • Arganda-Carreras I.
      • Turaga S.C.
      • Berger D.R.
      • Cireşan D.
      • Giusti A.
      • Gambardella L.M.
      • et al.
      Crowdsourcing the creation of image segmentation algorithms for connectomics.
      ,
      • Kwon J.M.
      • Jeon K.H.
      • Kim H.M.
      • Kim M.J.
      • Lim S.
      • Kim K.H.
      • et al.
      Deep-learning-based risk stratification for mortality of patients with acute myocardial infarction.
      ,
      • Tomita K.
      • Nagao R.
      • Touge H.
      • Ikeuchi T.
      • Sano H.
      • Yamasaki A.
      • et al.
      Deep learning facilitates the diagnosis of adult asthma.
      ,
      • Bahado-Singh R.O.
      • Sonek J.
      • McKenna D.
      • Cool D.
      • Aydas B.
      • Turkoglu O.
      • et al.
      Artificial intelligence and amniotic fluid multiomics: prediction of perinatal outcome in asymptomatic women with short cervix.
      ,
      • Zhang Q.
      • Liu Y.
      • Liu G.
      • Zhao G.
      • Qu Z.
      • Yang W.
      An automatic diagnostic system based on deep learning, to diagnose hyperlipidemia.
      ,
      • Gupta A.
      • Müller A.T.
      • Huisman B.J.H.
      • Fuchs J.A.
      • Schneider P.
      • Schneider G.
      Generative Recurrent Networks For De Novo Drug Design.
      ,
      • Merk D.
      • Friedrich L.
      • Grisoni F.
      • Schneider G.
      De Novo design of bioactive small molecules by artificial intelligence.
      ,
      • Mamoshina P.
      • Volosnikova M.
      • Ozerov I.V.
      • Putin E.
      • Skibina E.
      • Cortese F.
      • et al.
      Machine learning on human muscle transcriptomic data for biomarker discovery and tissue-specific drug target identification.
      ,
      • Su X.
      • Xu J.
      • Yin Y.
      • Quan X.
      • Zhang H.
      Antimicrobial peptide identification using multi-scale convolutional network.
      ,
      • Zhang Y.Z.
      • Yamaguchi R.
      • Imoto S.
      • Miyano S.
      Sequence-specific bias correction for RNA-seq data using recurrent neural networks.
      ,
      • Wang Y.
      • Zhang S.
      • Yang L.
      • Yang S.
      • Tian Y.
      • Ma Q.
      Measurement of conditional relatedness between genes using fully convolutional neural network.
      ,
      • Mamoshina P.
      • Kochetov K.
      • Cortese F.
      • Kovalchuk A.
      • Aliper A.
      • Putin E.
      • et al.
      Blood biochemistry analysis to detect smoking status and quantify accelerated aging in smokers.
      ,
      • Kanda E.
      • Epureanu B.I.
      • Adachi T.
      • Tsuruta Y.
      • Kikuchi K.
      • Kashihara N.
      • et al.
      Application of explainable ensemble artificial intelligence model to categorization of hemodialysis-patient and treatment using nationwide-real-world data in Japan.
      ,
      • Falissard L.
      • Morgand C.
      • Roussel S.
      • Imbaud C.
      • Ghosn W.
      • Bounebache K.
      • et al.
      A deep artificial neural network-based model for prediction of underlying cause of death from death certificates: algorithm development and validation.
      ,
      • Abdulaal A.
      • Patel A.
      • Charani E.
      • Denny S.
      • Mughal N.
      • Moore L.
      Prognostic modeling of COVID-19 using artificial intelligence in the United Kingdom: model development and validation.
      ,
      • Tucker B.A.
      • Mullins R.F.
      • Stone E.M.
      Autologous cell replacement: a noninvasive AI approach to clinical release testing.
      ,
      • Chishti S.
      • Jaggi K.R.
      • Saini A.
      • Agarwal G.
      • Ranjan A.
      Artificial intelligence-based differential diagnosis: development and validation of a probabilistic model to address lack of large-scale clinical datasets.
      ,
      • Smolander J.
      • Dehmer M.
      • Emmert-Streib F.
      Comparing deep belief networks with support vector machines for classifying gene expression data from complex disorders.
      ,
      • Kadurin A.
      • Aliper A.
      • Kazennov A.
      • Mamoshina P.
      • Vanhaelen Q.
      • Khrabrov K.
      • et al.
      The cornucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology.
      ,
      • Krsnik I.
      • Glavaš G.
      • Krsnik M.
      • Miletić D.
      • Štajduhar I.
      Automatic Annotation of Narrative Radiology Reports.
      ,
      • He X.
      • Zhao L.
      • Zhong W.
      • Chen H.Y.
      • Shan X.
      • Tang N.
      • et al.
      Insight into potent leads for alzheimer's disease by using several artificial intelligence algorithms.
      ,
      • Liu Y.
      • Palmedo P.
      • Ye Q.
      • Berger B.
      • Peng J.
      Enhancing evolutionary couplings with deep convolutional neural networks.
      ,
      • Engle E.
      • Gabrielian A.
      • Long A.
      • Hurt D.E.
      • Rosenthal A.
      Performance of Qure.ai automatic classifiers against a large annotated database of patients with diverse forms of tuberculosis.
      ,
      • Zhang L.
      • Qu Y.
      • Jin B.
      • Jing L.
      • Gao Z.
      • Liang Z.
      An intelligent mobile-enabled system for diagnosing Parkinson disease: development and validation of a speech impairment detection system.
      ,
      • Benke K.
      • Benke G.
      Artificial intelligence and big data in public health.
      ,
      • Kriegeskorte N.
      • Golan T.
      Neural network models and deep learning.
      ,
      • Signaevsky M.
      • Prastawa M.
      • Farrell K.
      • Tabish N.
      • Baldwin E.
      • Han N.
      • et al.
      Artificial intelligence in neuropathology: deep learning-based assessment of tauopathy.
      ,
      • Park S.H.
      • Do K.H.
      • Kim S.
      • Park J.H.
      • Lim Y.S.
      What should medical students know about artificial intelligence in medicine?.
      ,
      • de Lange T.
      • Halvorsen P.
      • Riegler M.
      Methodology to develop machine learning algorithms to improve performance in gastrointestinal endoscopy.
      ,
      • Zhu J.S.
      • Ge P.
      • Jiang C.
      • Zhang Y.
      • Li X.
      • Zhao Z.
      • et al.
      Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients.
      ,
      • Paranjape K.
      • Schinkel M.
      • Nannan Panday R.
      • Car J.
      • Nanayakkara P.
      Introducing artificial intelligence training in medical education.
      ,
      • Hammad I.
      • El-Sankary K.
      Practical considerations for accuracy evaluation in sensor-based machine learning and deep learning.
      ,
      • Bhhatarai B.
      • Walters W.P.
      • Hop C.
      • Lanza G.
      • Ekins S.
      Opportunities and challenges using artificial intelligence in ADME/Tox.
      ,
      • Carpenter K.A.
      • Cohen D.S.
      • Jarrell J.T.
      • Huang X.
      Deep learning and virtual drug screening.
      ,
      • Kulikowski C.A.
      Beginnings of artificial intelligence in medicine (AIM): computational artifice assisting scientific inquiry and clinical art - with reflections on present AIM challenges.
      ,
      • Li J.
      • Xu Q.
      • Shah N.
      • Mackey T.K.
      A machine learning approach for the detection and characterization of illicit drug dealers on instagram: model evaluation study.
      ,
      • Cifuentes-Alcobendas G.
      • Domínguez-Rodrigo M.
      Deep learning and taphonomy: high accuracy in the classification of cut marks made on fleshed and defleshed bones using convolutional neural networks.
      ,
      • Abbod M.
      • Shieh J.S.
      Advanced signal processing in intelligent systems for health monitoring.
      ,
      • Guerguiev J.
      • Lillicrap T.P.
      • Richards B.A.
      Towards deep learning with segregated dendrites.
      ,
      • Zhang J.
      • Su Q.
      • Loudon W.G.
      • Lee K.L.
      • Luo J.
      • Dethlefs B.A.
      • et al.
      Breathing signature as vitality score index created by exercises of qigong: implications of artificial intelligence tools used in traditional Chinese medicine.
      ,
      • Norouzzadeh M.S.
      • Nguyen A.
      • Kosmala M.
      • Swanson A.
      • Palmer M.S.
      • Packer C.
      • et al.
      Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning.
      ,
      • Sardi S.
      • Vardi R.
      • Goldental A.
      • Sheinin A.
      • Uzan H.
      • Kanter I.
      Adaptive nodes enrich nonlinear cooperative learning beyond traditional adaptation by links.
      ,
      • Ting D.S.
      • Rim T.H.
      • Choi Y.S.
      • Ledsam J.R.
      Deep learning in medicine. Are we ready?.
      ,
      • Tobore I.
      • Li J.
      • Yuhang L.
      • Al-Handarish Y.
      • Kandwal A.
      • Nie Z.
      • et al.
      Deep learning intervention for health care challenges: some biomedical domain considerations.
      ,
      • Jauregi Unanue I.
      • Zare Borzeshi E.
      • Piccardi M.
      Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition.
      ,
      • Feldmann J.
      • Youngblood N.
      • Wright C.D.
      • Bhaskaran H.
      • Pernice W.H.P.
      All-optical spiking neurosynaptic networks with self-learning capabilities.
      ,
      • Fiosina J.
      • Fiosins M.
      • Bonn S.
      Explainable deep learning for augmentation of small RNA expression profiles.
      ,
      • Yu H.
      • Samuels D.C.
      • Zhao Y.Y.
      • Guo Y.
      Architectures and accuracy of artificial neural network for disease classification from omics data.
      ,
      • Kiral-Kornek I.
      • Roy S.
      • Nurse E.
      • Mashford B.
      • Karoly P.
      • Carroll T.
      • et al.
      Epileptic seizure prediction using big data and deep learning: toward a mobile system.
      ,
      • Rere L.M.
      • Fanany M.I.
      • Arymurthy A.M.
      Metaheuristic algorithms for convolution neural network.
      ,
      • Kamel Boulos M.N.
      • Peng G.
      • VoPham T.
      An overview of GeoAI applications in health and healthcare.
      ,
      • Tran D.Q.
      • Kim J.W.
      • Tola K.D.
      • Kim W.
      • Park S.
      Artificial intelligence-based bolt loosening diagnosis using deep learning algorithms for laser ultrasonic wave propagation data.
      ,
      • Schulz M.A.
      • Yeo B.T.T.
      • Vogelstein J.T.
      • Mourao-Miranada J.
      • Kather J.N.
      • Kording K.
      • et al.
      Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets.
      ,
      • Gebru T.
      • Krause J.
      • Wang Y.
      • Chen D.
      • Deng J.
      • Aiden E.L.
      • et al.
      Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States.
      ,
      • Tedeschi L.O.
      ASN-ASAS SYMPOSIUM: FUTURE OF DATA ANALYTICS IN NUTRITION: Mathematical modeling in ruminant nutrition: approaches and paradigms, extant models, and thoughts for upcoming predictive analytics1,2.
      ,
      • Yakimovich A.
      mSphere of influence: the rise of artificial intelligence in infection biology.
      ,
      • Tong F.
      • Shahid M.
      • Jin P.
      • Jung S.
      • Kim W.H.
      • Kim J.
      Classification of the urinary metabolome using machine learning and potential applications to diagnosing interstitial cystitis.
      ,
      • Sevillano V.
      • Aznarte J.L.
      Improving classification of pollen grain images of the POLEN23E dataset through three different applications of deep learning convolutional neural networks.
      ,
      • Gomez J.
      • Doukky R.
      Artificial intelligence in nuclear cardiology.
      ,
      • Matta C.F.
      • Hutter M.C.
      Foreword: computational special focus issue.
      ,
      • Unberath M.
      • Zaech J.N.
      • Gao C.
      • Bier B.
      • Goldmann F.
      • Lee S.C.
      • et al.
      Enabling machine learning in X-ray-based procedures via realistic simulation of image formation.
      ,
      • Lopez-Vazquez V.
      • Lopez-Guede J.M.
      • Marini S.
      • Fanelli E.
      • Johnsen E.
      • Aguzzi J.
      Video image enhancement and machine learning pipeline for underwater animal detection and classification at cabled observatories.
      ,
      • Liang J.
      • Xu Y.
      • Liu R.
      • Zhu X.
      QM-sym, a symmetrized quantum chemistry database of 135 kilo molecules.
      ,
      • Clark R.D.
      Putting deep learning in perspective for pest management scientists.
      ,
      • Zawbaa H.M.
      • Szlȩk J.
      • Grosan C.
      • Jachowicz R.
      • Mendyk A.
      Computational intelligence modeling of the macromolecules release from PLGA microspheres-focus on feature selection.
      ,
      • Bezemer T.
      • de Groot M.C.
      • Blasse E.
      • Ten Berg M.J.
      • Kappen T.H.
      • Bredenoord A.L.
      • et al.
      A Human(e) factor in clinical decision support systems.
      ,
      • Carpenter K.
      • Huang X.
      Is it a prime time for AI-powered virtual drug screening?.
      ,
      • Babanezhad M.
      • Taghvaie Nakhjiri A.
      • Rezakazemi M.
      • Shirazian S.
      Developing intelligent algorithm as a machine learning overview over the big data generated by Euler-Euler method to simulate bubble column reactor hydrodynamics.
      ,
      • Choi J.
      • Shin K.
      • Jung J.
      • Bae H.J.
      • Kim D.H.
      • Byeon J.S.
      • et al.
      Convolutional neural network technology in endoscopic imaging: artificial intelligence for endoscopy.
      ,
      • Chatterjee A.
      • Gerdes M.W.
      • Martinez S.G.
      Statistical explorations and univariate timeseries analysis on COVID-19 datasets to understand the trend of disease spreading and death.
      ,
      • Olsavszky V.
      • Dosius M.
      • Vladescu C.
      • Benecke J.
      Time series analysis and forecasting with automated machine learning on a national ICD-10 database.
      ,
      • Luo S.
      • Leung A.P.
      • Qiu X.
      • Chan J.Y.K.
      • Huang H.
      complementary deep and shallow learning with boosting for public transportation safety.
      ,
      • Yang Z.
      • Dehmer M.
      • Yli-Harja O.
      • Emmert-Streib F.
      Combining deep learning with token selection for patient phenotyping from electronic health records.
      ,
      • Pikhart M.
      Intelligent information processing for language education: the use of artificial intelligence in language learning apps.
      ,
      • Anvarjon T.
      • Mustaqeem K.S.
      Deep-Net: a lightweight CNN-based speech emotion recognition system using deep frequency features.
      ,
      • Asgher U.
      • Khalil K.
      • Khan M.J.
      • Ahmad R.
      • Butt S.I.
      • Ayaz Y.
      • et al.
      Enhanced accuracy for multiclass mental workload detection using long short-term memory for brain-computer interface.
      ,
      • Guo S.
      • Xu P.
      • Miao Q.
      • Shao G.
      • Chapman C.A.
      • Chen X.
      • et al.
      Automatic identification of individual primates with deep learning techniques.
      ,
      • Baselli G.
      • Codari M.
      • Sardanelli F.
      Opening the black box of machine learning in radiology: can the proximity of annotated cases be a way?.
      ,
      • Nind T.
      • Sutherland J.
      • McAllister G.
      • Hardy D.
      • Hume A.
      • MacLeod R.
      • et al.
      An extensible big data software architecture managing a research resource of real-world clinical radiology data linked to other health data from the whole Scottish population.
      ,
      • Shateri M.
      • Sobhanigavgani Z.
      • Alinasab A.
      • Varamesh A.
      • Hemmati-Sarapardeh A.
      • Mosavi A.
      • et al.
      Comparative analysis of machine learning models for nanofluids viscosity assessment.
      ,
      • Kakarmath S.
      • Esteva A.
      • Arnaout R.
      • Harvey H.
      • Kumar S.
      • Muse E.
      • et al.
      Best practices for authors of healthcare-related artificial intelligence manuscripts.
      ,
      • Trivedi H.M.
      • Panahiazar M.
      • Liang A.
      • Lituiev D.
      • Chang P.
      • Sohn J.H.
      • et al.
      Large scale semi-automated labeling of routine free-text clinical records for deep learning.
      ] screened were excluded for the following reasons: Not inherent (# 55) and Overview/Summary (# 27). Out of the 87 full-text articles assessed for eligibility, 5full-text articles were excluded with reasons(i.e. few patients or not images). Out of 82 included papers (reported in supplementary material Table S1), the following subgroups methods were obtained: Artificial Intelligence (# 6), Deep Learning (# 65) or Machine learning (# 11). The excluded papers are reported in supplementary material Table S2.
      Figure thumbnail gr2
      Fig. 2PRISMA Flow-Diagram of study selection and inclusion in the systematic review.
      The distribution of papers according to the discipline and type or algorithm is reported in Fig. 3. Out of 65 paper focusing on DL, the majority regarded the radiology (# 8), oncology (# 12), neurology (# 7), cardiology (# 6) and biology (# 6) field. Out 7 papers on AI, the radiology and oncology field were investigated in 2 studies. Out of 10 papers using ML, # 3 focused on the field of biology (for more details see Fig. 3).
      Figure thumbnail gr3
      Fig. 3The distribution of investigated papers according the type or algorithm and the discipline.
      The main aim of the included manuscripts can be summarized as follows: classification (# 40), detection (# 37), segmentation (# 9), predict outcomes (# 12). A combination of them, to the used algorithm type, was reported in Fig. 4.
      Figure thumbnail gr4
      Fig. 4The distribution of investigated papers according to the main aim of the paper.

      Image type

      Forty-six of the 82 included papers are related to imaging analysis belonging to several fields of medicine. The medical image types investigated can be clustered in different area groups: AI (# 5), DL (# 37), ML (# 4). The types of investigated images are reported in supplementary material Table S3.

      Training, validation and test set

      A robust application of AI strategies in medical studies requires a training, validation and test phases. The training dataset is a set of data used to fit the parameters of the model such as weights of connections between neurons. The training dataset often consists of pairs of an input vector (or scalar) and the corresponding output vector (or scalar). The model result is compared with the expected one to adjust the model parameters. Each iteration calculates the difference between the predicted and actual outcomes and refines the algorithms weightings accordingly while reducing this difference. The produced algorithm is tailored specifically to the training data set. To optimize the performance of the algorithm, assess its generalizability of the final algorithm and its learnt parameters, the model is applied on the test set, also used for tuning hyperparameters. The validation dataset is data used to evaluate the model on a “new” external set of data.
      The number of hidden units, layers or layer width can for example be iteratively tuned stopping training when the error on the validation dataset increases, as this is a sign of over fitting to the training dataset. The test dataset provides finally an unbiased evaluation of a final model fit using a completely new set of data.
      As reported by Shahin et al. [
      • Shahin M.A.
      • Maier H.R.
      • Jaska M.B.
      Data division for developing neural networks applied to geotechnical engineering.
      ], the training dataset often consists of pairs of an input vector (or scalar) and the corresponding output vector (or scalar) and it is a set of data used to fit the parameters of the model such as weights of connections between neurons, for example. It is well-understood that the training set performance tends to overestimate the test set accuracy, therefore It is advantageous to use a test set to evaluate the trained model's performance during hyper-parameter search and model optimization. These are commonly called “Resampling methods” and one of the most popular method is “k-fold cross validation” or “leave-x-out cross validation” where the training set is divided in k parts and each of the n original cases has been left out exactly once (x = n/k). The resulting algorithm will then be tailored specifically to the training data. There are many possible ways to split the original dataset and this task could be dependent on data availability (e.g., 20% of dataset can be used for training). A split providing 20% validation and 30% test from the remaining 80% has been shown to produce good generalization from the test to validation set accuracy [
      • Shahin M.A.
      • Maier H.R.
      • Jaska M.B.
      Data division for developing neural networks applied to geotechnical engineering.
      ].
      The mean value of percentage of patients/images used as training, test and validation dataset according to the different approach is shown in Fig. 5. Artificial Intelligence area includes studies in which algorithm is no better specified to belong at ML or DP area. The training set were 72%, 73%, 78% for AI, DP and ML, respectively. The test set were 7%, 20%, 17% for AI, DP and ML, respectively. Finally, 21%,7% and 5% were the percentage for validation test, respectively.
      Figure thumbnail gr5
      Fig. 5The mean value of percentage of patients/images used as training, test and validation dataset according to the different approach.
      The percentage of them, considering all the included articles, are: 71%, 6% and 23% for training, validation and test purposes respectively.
      Several tools were implemented for improving the accuracy, the efficiency and the performance of the models for detection/classification and outcome prediction. The input images ranged from CT and radiographs to dermoscopy, microscopy or spectroscopy. Our overview indicates that the amount of research on image-based AI models is not balanced across districts, as some of them, such as brain, breast, liver and lungs have been receiving more attention than other sites. Images were extracted from mono-institutional or multicentric database or free available datasets (e.g., Table 1).
      Table 1Examples of free available datasets used in the reviewed AI studies.
      Image type and additional informationAvailable database
      Digital MammogramThe mini‑MIAS
      • Trivizakis E.
      • Ioannidis G.S.
      • Melissianos V.D.
      • Papadakis G.Z.
      • Tsatsakis A.
      • Spandidos D.A.
      • et al.
      A novel deep learning architecture outperforming 'off-the-shelf' transfer learning and feature-based methods in the automated assessment of mammographic breast density.
      is a free scientific database for research and consists of 161 patients with 322 mammograms.
      The DDSM
      • Trivizakis E.
      • Ioannidis G.S.
      • Melissianos V.D.
      • Papadakis G.Z.
      • Tsatsakis A.
      • Spandidos D.A.
      • et al.
      A novel deep learning architecture outperforming 'off-the-shelf' transfer learning and feature-based methods in the automated assessment of mammographic breast density.
      database consists of approximately 2500 patients with 10,239 multi‑view images including benign, malignant and normal cases.
      The Breast Cancer Wisconsin (BCW) dataset includes 683 cases (after removing 16 cases with missing values), 9 dimensions with integer values ranging from 0 to 10 (computed from a digitized image of fine needle aspirate of breast mass) and 2 classes (whether the diagnostic is benign or malignant)
      • Lamy J.B.
      • Sekar B.
      • Guezennec G.
      • Bouaud J.
      • Séroussi B.
      Explainable artificial intelligence for breast cancer: a visual case-based reasoning approach.
      .
      The Mammographic Mass (MM) dataset includes 830 cases, 2 numeric dimensions (age and Breast Imaging Reporting And Data System value, BI-RADS), 3 categorical dimensions (shape, margin and density of the mass) and 2 classes (whether the diagnostic is benign or malignant)
      • Choudhury N.
      • Begum S.
      A survey on case-based reasoning in medicine.
      .
      The Breast Cancer (BC) dataset includes 286 cases, 4 numeric dimensions (age, tumor size, etc.), 4 categorical dimensions (breast quadrant, etc.) and 2 classes (whether the cancer is recurrent or not)
      • Damiani G.
      • Pinnarelli L.
      • Colosimo S.C.
      • Almiento R.
      • Sicuro L.
      • Galasso R.
      • et al.
      The effectiveness of computerized clinical guidelines in the process of care: a systematic review.
      .
      Dermatoscopic imagesBiopsy verified skin lesion image set from the International Skin Imaging Collaboration (ISIC) archive
      • VoPham T.
      • Hart J.E.
      • Laden F.
      • Chiang Y.Y.
      Emerging trends in geospatial artificial intelligence (geoAI): potential applications for environmental epidemiology.
      .
      HAM10000 Dataset contains dermoscopic images of heterogeneous populations that are publicly accessible, anonymous and taken by different camera systems and thus have a high external validity
      • Tschandl P.
      • Rosendahl C.
      • Kittler H.
      The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.
      .
      MRIBRATS 2015 and BRATS 2016 brain tumor segmentations challenge datasets
      • Schmidhuber J.
      Deep learning in neural networks: an overview.
      ,
      • LeCun Y.
      • Bengio Y.
      • Hinton G.
      Deep learning.
      ,
      • Hoseini F.
      • Shahbahrami A.
      • Bayat P.
      AdaptAhead optimization algorithm for learning deep CNN applied to MRI segmentation.
      .
      The Autism Brain Imaging Data Exchange (ABIDE) repository contains pre-processed resting-state functional MRI (rs-fMRI) data from ASD (autism spectrum disorder) and healthy subjects across all independent data sites
      • Li H.
      • Parikh N.A.
      • He L.
      A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes.
      .
      Radiologic, molecular, genetic, pathological and clinical historyThe Cancer Genome Atlas’ (TCGA) includes hepatocellular carcinoma (LIHC) and cholangiocarcinoma(CHOL) diagnostic FFPE WSI collections.
      TCGA dataset metadata
      • Kiani A.
      • Uyumazturk B.
      • Rajpurkar P.
      • Wang A.
      • Gao R.
      • Jones E.
      • et al.
      Impact of a deep learning assistant on the histopathologic classification of liver cancer.
      available through the TCGA’s Genomic Data Commons (GDC), including the original pathology reports (PDF files) containing the results of any additional immune-histochemical or special stains performed, as well as available radiologic, molecular diagnostic, and clinical history
      The Cancer Genome Atlas (TCGA) Lower-Grade Glioma

      (LGG) and Glioblastoma (GBM) projects
      • Mobadersany P.
      • Yousefi S.
      • Amgad M.
      • Gutman D.A.
      • Barnholtz-Sloan J.S.
      • Velázquez Vega J.E.
      • et al.
      Predicting cancer outcomes from histology and genomics using convolutional networks.
      .
      TCGA’s dataset
      • Grossman R.L.
      • Heath A.P.
      • Ferretti V.
      • Varmus H.E.
      • Lowy D.R.
      • Kibbe W.A.
      • et al.
      Toward a shared vision for cancer genomic data.
      .
      SpectrometryMETASPACE datasets
      • Ovchinnikova K.
      • Kovalev V.
      • Stuart L.
      • Alexandrov T.
      OffsampleAI: artificial intelligence approach to recognize off-sample mass spectrometry images.
      .
      XrayGitHub covid-chest-X-ray dataset
      • Borkowski A.A.
      • Viswanadhan N.A.
      • Thomas L.B.
      • Guzman R.D.
      • Deland L.A.
      • Mastorides S.M.
      Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis.
      .
      ImageNet

      Deng J, Dong W, Socher R, Li L, Kai L, Li F-F, editors. ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition; 2009 20-25 June 2009.

      .
      EEGImageNet – EEG

      Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M, editors. Deep Learning Human Mind for Automated Visual Classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 21-26 July 2017.

      ,
      • Fares A.
      • Zhong S.H.
      • Jiang J.
      EEG-based image classification via a region-level stacked bi-directional deep learning framework.
      .
      Neuroimagingthe Alzheimer’s Disease Neuroimaging Initiative (ADNI) database
      • Huang Y.
      • Xu J.
      • Zhou Y.
      • Tong T.
      • Zhuang X.
      Diagnosis of Alzheimer's disease via multi-modality 3D convolutional neural network.
      and the 273 regions designated in the Brainnetome Atlas
      • Jin D.
      • Zhou B.
      • Han Y.
      • Ren J.
      • Han T.
      • Liu B.
      • et al.
      Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer's disease.
      .
      Several codes according to the main aim have been implemented/used as summarized in Table S3 (supplementary data) according to the main aim of papers. Of note, only a limited part of the reported software has been compared one each other within the indicated studies.
      The most frequently used AI Network architectures [
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ]:
      • A DNN is composed of several hidden layers in which all neurons of a layer are connected to all the neurons of the following layer. The potential disadvantage is a slow learning process.
      • A recurrent neural network (RNN) is useful for the processing of temporally dependent information. The output of the network is a function not only of the input at a specific time but also of the inputs at previous times.
      • An auto-encoder (AE) is composed of a hidden layer that encodes layer for identifying a latent dominant structure of reduced size with respect to the input signal. The architecture is characterized by a full connection between neurons in input, hidden and output layers. Several variants have been developed; one of the advantages s that it does not require labeled data using unsupervised learning.
      • The Boltzmann machine (BM) is a model in which all the neurons are bidirectionally connected to each other; it can generate new input data during the unsupervised learning.
      • In Deep belief network (DBNs) only the two deepest layers have bidirectional connections. The training is unsupervised, except for the final adjustment of the network parameters performed in a supervised approach by adding a classification layer at the output of the network.
      • The generative adversarial networks (GANs) are based on two models: a generative model that produces synthetic data, and a discriminating model that estimates the probability that these data are part of the training data
      • CNN consists of several successive layers of data processing to find representative features of the input image, then more elaborate as the layers succeed each other. CNNs can be learned using both unsupervised or supervised approaches. CNNs are specifically designed to take advantage of spatially structured information such as medical images in which neighboring pixels corresponding to the same anatomical structure could share similar intensity characteristics; a specifically parameterized patch of neurons might detect pixels corresponding to the same anatomical region.

      Application of AI tools

      This section presents the evolution of the contributions of AI according to the different application areas. The supplementary material for this article includes a list of the contributions of the AI techniques applied on disciplines listed in Fig. 3 considering various typologies of images.

      Biology

      AI tools have been used in Classification/Detection or in outcomes prediction and among other for modeling of biological structures [
      • Lawrimore J.
      • Doshi A.
      • Walker B.
      • Bloom K.
      AI-assisted forward modeling of biological structures.
      ,
      • Kusumoto D.
      • Lachmann M.
      • Kunihiro T.
      • Yuasa S.
      • Kishino Y.
      • Kimura M.
      • et al.
      Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells.
      ,
      • Fischer C.A.
      • Besora-Casals L.
      • Rolland S.G.
      • Haeussler S.
      • Singh K.
      • Duchen M.
      • et al.
      MitoSegNet: easy-to-use deep learning segmentation for analyzing mitochondrial morphology.
      ], prediction and characterization of specific sequences [
      • Zhang Y.
      • Hamada M.
      DeepM6ASeq: prediction and characterization of m6A-containing sequences using deep learning.
      ,
      • Yamada K.D.
      • Kinoshita K.
      De novo profile generation based on sequence context specificity with the long short-term memory network.
      ] or proteins [
      • Ho Thanh Lam L.
      • Le N.H.
      • Van Tuan L.
      • Tran Ban H.
      • Nguyen Khanh Hung T.
      • Nguyen N.T.K.
      • et al.
      Machine learning model for identifying antioxidant proteins using features calculated from primary sequences.
      ], recognize off-sample mass spectrometry images [
      • Ovchinnikova K.
      • Kovalev V.
      • Stuart L.
      • Alexandrov T.
      OffsampleAI: artificial intelligence approach to recognize off-sample mass spectrometry images.
      ], performing dynamic measurement of single-cell volume [
      • Yao K.
      • Rochman N.D.
      • Sun S.X.
      CTRL - a label-free artificial intelligence method for dynamic measurement of single-cell volume.
      ] or classification of lung cancer patients based on mRNA and non-coding RNAs [
      • Smolander J.
      • Stupnikov A.
      • Glazko G.
      • Dehmer M.
      • Emmert-Streib F.
      Comparing biological information contained in mRNA and non-coding RNAs for classification of lung cancer patients.
      ].

      Cardiology

      In cardiology, deep learning algorithms were used for predicting/identify heart failure [
      • Kwon J.M.
      • Kim K.H.
      • Jeon K.H.
      • Kim H.M.
      • Kim M.J.
      • Lim S.M.
      • et al.
      Development and validation of deep-learning algorithm for electrocardiography-based heart failure identification.
      ,
      • Kwon J.M.
      • Kim K.H.
      • Jeon K.H.
      • Lee S.E.
      • Lee H.Y.
      • Cho H.J.
      • et al.
      Artificial intelligence algorithm for predicting mortality of patients with acute heart failure.
      ,
      • Kwon J.M.
      • Lee Y.
      • Lee Y.
      • Lee S.
      • Park J.
      An Algorithm based on deep learning for predicting in-hospital cardiac arrest.
      ], assessing patients with a systemic right ventricle [
      • Diller G.P.
      • Babu-Narayan S.
      • Li W.
      • Radojevic J.
      • Kempny A.
      • Uebing A.
      • et al.
      Utility of machine learning algorithms in assessing patients with a systemic right ventricle.
      ], detecting hypokalemia and hyperkalemia by electrocardiography: algorithm development [
      • Lin C.S.
      • Lin C.
      • Fang W.H.
      • Hsu C.J.
      • Chen S.J.
      • Huang K.H.
      • et al.
      A Deep-Learning Algorithm (ECG12Net) for detecting hypokalemia and hyperkalemia by electrocardiography: algorithm development.
      ], or the dofetilide plasma concentration based on of the surface electrocardiogram [
      • Attia Z.I.
      • Sugrue A.
      • Asirvatham S.J.
      • Ackerman M.J.
      • Kapa S.
      • Friedman P.A.
      • et al.
      Noninvasive assessment of dofetilide plasma concentration using a deep learning (neural network) analysis of the surface electrocardiogram: a proof of concept study.
      ].

      Dermatology

      In dermatology, the recognition of visual patterns is fundamental for skin diagnosis. AI tools are expected increasing the sensitivity and specificity in classifying images when combined with dermatologist’ expertise. Recently, whole-slide imaging now enables dermatology to storage a vast amounts of images [
      • Hart S.N.
      • Flotte W.
      • Norgan A.P.
      • Shah K.K.
      • Buchan Z.R.
      • Mounajjed T.
      • et al.
      Classification of melanocytic lesions in selected and whole-slide images via convolutional neural networks.
      ]. Artificial Intelligence or Deep Learning methods were used for skin cancer classification/diagnosis [
      • Hekler A.
      • Utikal J.S.
      • Enk A.H.
      • Hauschild A.
      • Weichenthal M.
      • Maron R.C.
      • et al.
      Superior skin cancer classification by the combination of human and artificial intelligence.
      ,
      • Walker B.N.
      • Rehg J.M.
      • Kalra A.
      • Winters R.M.
      • Drews P.
      • Dascalu J.
      • et al.
      Dermoscopy diagnosis of cancerous lesions utilizing dual deep learning algorithms via visual and audio (sonification) outputs: laboratory and prospective observational studies.
      ] melanoma image classification [
      • Maron R.C.
      • Utikal J.S.
      • Hekler A.
      • Hauschild A.
      • Sattler E.
      • Sondermann W.
      • et al.
      Artificial intelligence and its effect on dermatologists' accuracy in dermoscopic melanoma image classification: web-based survey study.
      ], classification of melanocytic [
      • Hart S.N.
      • Flotte W.
      • Norgan A.P.
      • Shah K.K.
      • Buchan Z.R.
      • Mounajjed T.
      • et al.
      Classification of melanocytic lesions in selected and whole-slide images via convolutional neural networks.
      ], onychomycosis diagnosis [
      • Han S.S.
      • Park G.H.
      • Lim W.
      • Kim M.S.
      • Na J.I.
      • Park I.
      • et al.
      Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
      ], classifying tissues [
      • Bizzego A.
      • Bussola N.
      • Chierici M.
      • Maggio V.
      • Francescatto M.
      • Cima L.
      • et al.
      Evaluating reproducibility of AI algorithms in digital pathology with DAPPER.
      ].

      Histopathology

      Large database of histopathology slides enable in depth analysis through deep learning analysis [
      • Jones A.D.
      • Graff J.P.
      • Darrow M.
      • Borowsky A.
      • Olson K.A.
      • Gandour-Edwards R.
      • et al.
      Impact of pre-analytical variables on deep learning accuracy in histopathology.
      ] or evaluating reproducibility of AI algorithms [
      • Bizzego A.
      • Bussola N.
      • Chierici M.
      • Maggio V.
      • Francescatto M.
      • Cima L.
      • et al.
      Evaluating reproducibility of AI algorithms in digital pathology with DAPPER.
      ], predicting cancer outcomes [
      • Mobadersany P.
      • Yousefi S.
      • Amgad M.
      • Gutman D.A.
      • Barnholtz-Sloan J.S.
      • Velázquez Vega J.E.
      • et al.
      Predicting cancer outcomes from histology and genomics using convolutional networks.
      ], detection or classification of Lymph Node Metastases [
      • Ehteshami Bejnordi B.
      • Veta M.
      • Johannes van Diest P.
      • van Ginneken B.
      • Karssemeijer N.
      • Litjens G.
      • et al.
      Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
      ,
      • Palatnik de Sousa I.
      • Maria Bernardes Rebuzzi Vellasco M.
      • Costa da Silva E.
      Local interpretable model-agnostic explanations for classification of lymph node metastases.
      ], histopathologic classification of liver cancer [
      • Kiani A.
      • Uyumazturk B.
      • Rajpurkar P.
      • Wang A.
      • Gao R.
      • Jones E.
      • et al.
      Impact of a deep learning assistant on the histopathologic classification of liver cancer.
      ] or facilitate the diagnosis of HER2 status in breast cancer [
      • Vandenberghe M.E.
      • Scott M.L.
      • Scorer P.W.
      • Söderberg M.
      • Balcerzak D.
      • Barker C.
      Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer.
      ].

      Embryology

      In the Embryology field, AI/Deep learning methods were used for the robust selection for prediction of embryo viability [
      • Khosravi P.
      • Kazemi E.
      • Zhan Q.
      • Malmsten J.E.
      • Toschi M.
      • Zisimopoulos P.
      • et al.
      Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization.
      ,
      • VerMilyea M.
      • Hall J.M.M.
      • Diakiw S.M.
      • Johnston A.
      • Nguyen T.
      • Perugini D.
      • et al.
      Development of an artificial intelligence-based assessment model for prediction of embryo viability using static images captured by optical light microscopy during IVF.
      ].

      Endocrinology

      Regarding the endocrinology field, AI methods improved staging of diabetic retinopathy [
      • Takahashi H.
      • Tampo H.
      • Arai Y.
      • Inoue Y.
      • Kawashima H.
      Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy.
      ,
      • Heydon P.
      • Egan C.
      • Bolter L.
      • Chambers R.
      • Anderson J.
      • Aldington S.
      • et al.
      Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients.
      ,
      • Shah P.
      • Mishra D.K.
      • Shanmugam M.P.
      • Doshi B.
      • Jayaraj H.
      • Ramanjulu R.
      Validation of Deep Convolutional Neural Network-based algorithm for detection of diabetic retinopathy - Artificial intelligence versus clinician for screening.
      ] or the progression of diabetic kidney disease [
      • Makino M.
      • Yoshimoto R.
      • Ono M.
      • Itoko T.
      • Katsuki T.
      • Koseki A.
      • et al.
      Artificial intelligence predicts the progression of diabetic kidney disease using big data machine learning.
      ] or improved the classification of gait biomarkers [
      • Sánchez-DelaCruz E.
      • Weber R.
      • Biswal R.R.
      • Mejía J.
      • Hernández-Chan G.
      • Gómez-Pozos H.
      Gait biomarkers classification by combining assembled algorithms and deep learning: results of a local study.
      ].

      Radiology

      Regarding the field of radiology, several type of images were investigated (including radiographs, screening mammograms, CT images, MRI, Computed tomography pulmonary angiography) for the automatic determination of the need for intravenous contrast in musculoskeletal MRI [
      • Trivedi H.
      • Mesterhazy J.
      • Laguna B.
      • Vu T.
      • Sohn J.H.
      Automatic determination of the need for intravenous contrast in musculoskeletal MRI examinations Using IBM Watson's natural language processing algorithm.
      ], interpretation of Screening Mammograms [
      • Schaffter T.
      • Buist D.S.M.
      • Lee C.I.
      • Nikulin Y.
      • Ribli D.
      • Guan Y.
      • et al.
      Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.
      ], for body morphometric analysis [
      • Lee H.
      • Troschel F.M.
      • Tajmir S.
      • Fuchs G.
      • Mario J.
      • Fintelmann F.J.
      • et al.
      Pixel-level deep segmentation: artificial intelligence quantifies muscle on computed tomography for body morphometric analysis.
      ], identifying cholelithiasis and classifying gallstones on CT images [
      • Pang S.
      • Ding T.
      • Qiao S.
      • Meng F.
      • Wang S.
      • Li P.
      • et al.
      A novel YOLOv3-arch model for identifying cholelithiasis and classifying gallstones on CT images.
      ], tuberculosis screening [
      • Pasa F.
      • Golkov V.
      • Pfeiffer F.
      • Cremers D.
      • Pfeiffer D.
      Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization.
      ], detection/visualization of fractures [
      • Cheng C.T.
      • Ho T.Y.
      • Lee T.Y.
      • Chang C.C.
      • Chou C.C.
      • Chen C.C.
      • et al.
      Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs.
      ,
      • Almog Y.A.
      • Rai A.
      • Zhang P.
      • Moulaison A.
      • Powell R.
      • Mishra A.
      • et al.
      Deep learning with electronic health records for short-term fracture risk identification: crystal bone algorithm development and validation.
      ], assessment for bone age [
      • Hu T.H.
      • Huo Z.
      • Liu T.A.
      • Wang F.
      • Wan L.
      • Wang M.W.
      • et al.
      Automated assessment for bone age of left wrist joint in uyghur teenagers by deep learning.
      ,
      • Lee H.
      • Tajmir S.
      • Lee J.
      • Zissen M.
      • Yeshiwas B.A.
      • Alkasab T.K.
      • et al.
      Fully automated deep learning system for bone age assessment.
      ] and sexual dimorphism in hand and wrist radiographs [
      • Yune S.
      • Lee H.
      • Kim M.
      • Tajmir S.H.
      • Gee M.S.
      • Do S.
      Beyond human perception: sexual dimorphism in hand and wrist radiographs is discernible by a deep learning model.
      ] or brain segmentation [
      • Bangalore Yogananda C.G.
      • Shah B.R.
      • Vejdani-Jahromi M.
      • Nalawade S.S.
      • Murugesan G.K.
      • Yu F.F.
      • et al.
      A fully automated deep learning network for brain tumor segmentation.
      ].
      Regarding the COVID-19 Infections, AI techniques have been used for its detection [6)], diagnosis using chest X-ray [
      • Borkowski A.A.
      • Viswanadhan N.A.
      • Thomas L.B.
      • Guzman R.D.
      • Deland L.A.
      • Mastorides S.M.
      Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis.
      ] or CT images [
      • Ardakani A.A.
      • Kanafi A.R.
      • Acharya U.R.
      • Khadem N.
      • Mohammadi A.
      Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: results of 10 convolutional neural networks.
      ].

      Ophthalmology

      Diabetic eye disease is amongst the most common conditions seen in routine ophthalmology practice and constitutes a significant and growing public health issue. Diabetic retinopathy is the commonest cause of vision loss in working age adults. The outcomes of these patients improve dramatically with early detection using digital imaging (high-resolution color retinal photography and optical coherence tomography). The utility of the AI algorithms to supplement clinician grading has been explored for diabetic retinal screening, diagnosis and appropriate referral of other sight threatening disorders, or replace current investigations. More in details, AI tools have been applied for identification of macular diseases or its degeneration [
      • Motozawa N.
      • An G.
      • Takagi S.
      • Kitahata S.
      • Mandai M.
      • Hirami Y.
      • et al.
      Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes.
      ,
      • Kuwayama S.
      • Ayatsuka Y.
      • Yanagisono D.
      • Uta T.
      • Usui H.
      • Kato A.
      • et al.
      Automated detection of macular diseases by optical coherence tomography and artificial intelligence machine learning of optical coherence tomography images.
      ,
      • Burlina P.M.
      • Joshi N.
      • Pekala M.
      • Pacheco K.D.
      • Freund D.E.
      • Bressler N.M.
      Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks.
      ]. Deep Learning Algorithms have been also used for glaucoma classification/detection [
      • Bajwa M.N.
      • Malik M.I.
      • Siddiqui S.A.
      • Dengel A.
      • Shafait F.
      • Neumeier W.
      • et al.
      Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning.
      ,
      • Christopher M.
      • Nakahara K.
      • Bowd C.
      • Proudfoot J.A.
      • Belghith A.
      • Goldbaum M.H.
      • et al.
      Effects of study population, labeling and training on glaucoma detection using deep learning algorithms.
      ,
      • Mariottoni E.B.
      • Datta S.
      • Dov D.
      • Jammal A.A.
      • Berchuck S.I.
      • Tavares I.M.
      • et al.
      Artificial intelligence mapping of structure to function in glaucoma.
      ].

      Neurology

      Regarding the neurology field, Deep Learning/Artificial Intelligence has been used for predicting cerebral palsy [
      • Bahado-Singh R.O.
      • Vishweswaraiah S.
      • Aydas B.
      • Mishra N.K.
      • Guda C.
      • Radhakrishna U.
      Deep learning/artificial intelligence and blood-based DNA Epigenomic prediction of cerebral palsy.
      ],the classification of brain functional Connectomes [
      • Li H.
      • Parikh N.A.
      • He L.
      A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes.
      ],diagnosis of Alzheimer's disease [
      • Fares A.
      • Zhong S.H.
      • Jiang J.
      EEG-based image classification via a region-level stacked bi-directional deep learning framework.
      ,
      • Jin D.
      • Zhou B.
      • Han Y.
      • Ren J.
      • Han T.
      • Liu B.
      • et al.
      Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer's disease.
      ], segmenting neuroanatomy [
      • Huang Y.
      • Xu J.
      • Zhou Y.
      • Tong T.
      • Zhuang X.
      Diagnosis of Alzheimer's disease via multi-modality 3D convolutional neural network.
      ] or diagnosing Parkinson disease through facial expression recognition [
      • Jin B.
      • Qu Y.
      • Zhang L.
      • Gao Z.
      Diagnosing Parkinson disease through facial expression recognition: video analysis.
      ].

      Pharmaceutics

      In the developing of pharmaceutic compounds, AI tools have been applied for prediction of pharmaceutical formulations [
      • Yang Y.
      • Ye Z.
      • Su Y.
      • Zhao Q.
      • Li X.
      • Ouyang D.
      Deep learning for in vitro prediction of pharmaceutical formulations.
      ] or prediction of adverse drug effects [
      • Gao M.
      • Igata H.
      • Takeuchi A.
      • Sato K.
      • Ikegaya Y.
      Machine learning-based prediction of adverse drug effects: an example of seizure-inducing compounds.
      ] or compare methods and metrics using diverse drug discovery data sets [
      • Korotcov A.
      • Tkachenko V.
      • Russo D.P.
      • Ekins S.
      Comparison of deep learning with multiple machine learning methods and metrics using diverse drug discovery data sets.
      ].

      Other fields

      Classifier algorithms for cross-person physical activity recognition [
      • Saez Y.
      • Baldominos A.
      • Isasi P.
      A comparison study of classifier algorithms for cross-person physical activity recognition.
      ], or fear level classification based on emotional dimensions or also building an otoscopic screening prototype [
      • Livingstone D.
      • Talai A.S.
      • Chau J.
      • Forkert N.D.
      Building an Otoscopic screening prototype tool using deep learning.
      ]. classifier algorithms have been also used for the classification of potential coronavirus treatments on a single human cell [
      • Khalifa N.E.M.
      • Taha M.H.N.
      • Manogaran G.
      • Loey M.
      A deep learning model and machine learning methods for the classification of potential coronavirus treatments on a single human cell.
      ]. regarding nephrology, deep learning algorithms has been also applied to grade hydronephrosis severity [
      • Smail L.C.
      • Dhindsa K.
      • Braga L.H.
      • Becker S.
      • Sonnadara R.R.
      Using deep learning algorithms to grade hydronephrosis severity: toward a clinical adjunct.
      ].

      Discussion

      The application of AI to imaging has generated significant interest within the medical disciplines such as radiology, pathology, ophthalmology, and dermatology. The selected papers, identified from the medical literature using the described search strategy, although there was some heterogeneity across included studies, were quite similar in terms of study design, methodology, analyses, and reporting, but present several limitations recognized by the Authors. In several studies [
      • Hekler A.
      • Utikal J.S.
      • Enk A.H.
      • Hauschild A.
      • Weichenthal M.
      • Maron R.C.
      • et al.
      Superior skin cancer classification by the combination of human and artificial intelligence.
      ,
      • Ehteshami Bejnordi B.
      • Veta M.
      • Johannes van Diest P.
      • van Ginneken B.
      • Karssemeijer N.
      • Litjens G.
      • et al.
      Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
      ,
      • Schaffter T.
      • Buist D.S.M.
      • Lee C.I.
      • Nikulin Y.
      • Ribli D.
      • Guan Y.
      • et al.
      Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.
      ] the data selection induced several biases. When only biopsy-verified images are considered, a certain bias is introduced, as these lesions are difficult to diagnose likely because during clinical examination of the patient, more information is available for the dermatologist for the diagnosis than just the visual impression of the examined skin area.
      In other cases [
      • Ehteshami Bejnordi B.
      • Veta M.
      • Johannes van Diest P.
      • van Ginneken B.
      • Karssemeijer N.
      • Litjens G.
      • et al.
      Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
      ], the test data was enriched with cases containing metastases and, specifically, micro metastases and, thus, is not directly comparable with the mix of cases pathologists encounter in clinical practice. In another, low-quality images and patients who had other concomitant diseases were excluded, so additional studies are necessary to ensure the utility of these methods in future clinical practice, such as telemedicine [
      • Motozawa N.
      • An G.
      • Takagi S.
      • Kitahata S.
      • Mandai M.
      • Hirami Y.
      • et al.
      Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes.
      ]. In addition, the importance of false negative findings for the patient outcome [
      • Takahashi H.
      • Tampo H.
      • Arai Y.
      • Inoue Y.
      • Kawashima H.
      Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy.
      ]. In principle, this information may also be taken into account by machine learning methods, leading to better classification quality in the future.
      In other studies [
      • Mobadersany P.
      • Yousefi S.
      • Amgad M.
      • Gutman D.A.
      • Barnholtz-Sloan J.S.
      • Velázquez Vega J.E.
      • et al.
      Predicting cancer outcomes from histology and genomics using convolutional networks.
      ,
      • Cheng C.T.
      • Ho T.Y.
      • Lee T.Y.
      • Chang C.C.
      • Chou C.C.
      • Chen C.C.
      • et al.
      Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs.
      ], a relatively small portion of each slide was used for training and prediction, and the selection of ROIs within each slide required expert guidance. if a certain proportion of images is manually excluded, this could be problematic when trying to incorporate this process into the clinical workflow using images from PACS in the hospital. Thus, future studies should explore more advanced methods for automatic selection of regions and for incorporating a higher proportion of each images in training and prediction to better account for image heterogeneity.
      The use of models based on training data which is not representative of the population, case mix, modalities, and acquisition protocols can compromise performance and confidence in its use, particularly if over-fitting has occurred [
      • Park S.H.
      • Han K.
      Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction.
      ]. Re-evaluations of algorithm performance in the clinical practice is a mandatory task and requires a practical understanding of these potential pitfalls to ensure the most responsible deployment of AI tools in practice.
      In addition, most studies were carried out in a single center with limited data availability, e.g. [
      • Wang C.J.
      • Hamm C.A.
      • Savic L.J.
      • Ferrante M.
      • Schobert I.
      • Schlachter T.
      • et al.
      Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features.
      ,
      • Zimmer D.
      • Schneider K.
      • Sommer F.
      • Schroda M.
      • Mühlhaus T.
      Artificial intelligence understands peptide observability and assists with absolute protein quantification.
      ,
      • Weng S.
      • Xu X.
      • Li J.
      • Wong S.T.C.
      Combining deep learning and coherent anti-Stokes Raman scattering imaging for automated differential diagnosis of lung cancer.
      ]. To avoid/reduce the over-fitting, the image set was augmented by randomly rotating the image several times and then flipping the rotated image horizontally [
      • Bahado-Singh R.O.
      • Vishweswaraiah S.
      • Aydas B.
      • Mishra N.K.
      • Guda C.
      • Radhakrishna U.
      Deep learning/artificial intelligence and blood-based DNA Epigenomic prediction of cerebral palsy.
      ] or free available image database have been used.
      In addition, most studies were retrospective, regards a short-term time frame [
      • Almog Y.A.
      • Rai A.
      • Zhang P.
      • Moulaison A.
      • Powell R.
      • Mishra A.
      • et al.
      Deep learning with electronic health records for short-term fracture risk identification: crystal bone algorithm development and validation.
      ] or covered a period of >10 years so the quality of data is unclear/questionable, and image quality varied between patients and across the study period [
      • Diller G.P.
      • Babu-Narayan S.
      • Li W.
      • Radojevic J.
      • Kempny A.
      • Uebing A.
      • et al.
      Utility of machine learning algorithms in assessing patients with a systemic right ventricle.
      ].
      AI tools are known as a “black box” because it is used to find the relationship between the input data and a result, not to create a rule based on knowledge and/or support the decision for a clinical endpoint/decision making [
      • Kwon J.M.
      • Kim K.H.
      • Jeon K.H.
      • Kim H.M.
      • Kim M.J.
      • Lim S.M.
      • et al.
      Development and validation of deep-learning algorithm for electrocardiography-based heart failure identification.
      ,
      • Kwon J.M.
      • Lee Y.
      • Lee Y.
      • Lee S.
      • Park J.
      An Algorithm based on deep learning for predicting in-hospital cardiac arrest.
      ]. Thus, interpretable DL needs to be implemented/developed.
      Of note, the Dialogue on Reverse Engineering Assessment and Methods (DREAM) Consortium has run dozens of biomedical challenges, establishing robust and objective computational benchmarks in multiple disease areas and across multiple data modalities [
      • Schaffter T.
      • Buist D.S.M.
      • Lee C.I.
      • Nikulin Y.
      • Ribli D.
      • Guan Y.
      • et al.
      Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.
      ,
      • Saez-Rodriguez J.
      • Costello J.C.
      • Friend S.H.
      • Kellen M.R.
      • Mangravite L.
      • Meyer P.
      • et al.
      Crowdsourcing biomedical research: leveraging communities as innovation engines.
      ], including the development and validation of breast cancer detection algorithms and the assessment of whether machine learning methods applied to mammography data can improve screening accuracy.
      Segmentation plays an important role in traditional classification methods and is also known to be time-consuming, so the possibility to extract useful features directly from raw images is challenging and of great interest, but currently limited by the calculation resources.
      To improve the replication of methods, several Authors freely disseminated the source codeor implemented websites (see Table S3) for demonstrating the potential of AI tools. At this stage only a limited part of the reported software/tools has been compared one each other. This can be considered an important improvement of the field. Open-source allows fostering future investigations on a massive scale. The most used open-source programming languages are currently Python and R, which can avail of libraries with implementations of the most widely used techniques and algorithms.
      Moreover, the performance of the algorithm was somewhat limited by heterogeneity in the training data [
      • Trivedi H.
      • Mesterhazy J.
      • Laguna B.
      • Vu T.
      • Sohn J.H.
      Automatic determination of the need for intravenous contrast in musculoskeletal MRI examinations Using IBM Watson's natural language processing algorithm.
      ]. In another study, performance of the proposed CAD system was not compared with radiologists [
      • Ardakani A.A.
      • Kanafi A.R.
      • Acharya U.R.
      • Khadem N.
      • Mohammadi A.
      Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: results of 10 convolutional neural networks.
      ].
      Within certain limitations and considering a purely image-based setting, artificial intelligence can achieve on par or superior performance to physician, thereby highlighting its potential as a decision-support system with immediate clinical implications [
      • Maron R.C.
      • Utikal J.S.
      • Hekler A.
      • Hauschild A.
      • Sattler E.
      • Sondermann W.
      • et al.
      Artificial intelligence and its effect on dermatologists' accuracy in dermoscopic melanoma image classification: web-based survey study.
      ]. In terms of data, AI efforts are expected to shift from processed medical images to raw acquisition data and unsupervised learning techniques to fully utilize all the available information.
      At the state-of-the-art AI is still in its infancy, and it is also evident that AI is unlikely to replace radiologists or other specialists but is expected to evolve into a valuable educational resource useful for discover hidden information that might have been overlooked. Initiatives are underway to standardize the requirements that must be met by an AI algorithm.
      Regarding the privacy, healthcare entities have been able to use and share de-identified records for research and development, either with patient consent or with waivers, and ensure the true de-identification or anonymization. For example, successful de-identification of medical imaging data stored in the DICOM format requires complete deletion or overwriting metadata. Another important limitation is the opacity and interpretability of these techniques used as ‘‘black boxes’’, without any biological explanation. Explainable AI represents the new frontiers for penetrating the “black box” of ML algorithms and build the basis of paradigm shifts in future image-based analysis in the daily clinical practice exceeding the cognitive bandwidth of human perception. The synergy of the human interpreter with the results of the artificial intelligence algorithm has not been fully addressed, nor how AI would affect the final assessments of radiologists [
      • Schaffter T.
      • Buist D.S.M.
      • Lee C.I.
      • Nikulin Y.
      • Ribli D.
      • Guan Y.
      • et al.
      Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.
      ]. This is an area that requires more research efforts.
      For these reasons, the application of AI on 2D/3D images continues to be an active research area for the application of precision medicine or research progresses of complex patterns from clinical databases.

      Future prospective

      In recent years, AI has been successfully used to solve real problems in a wide range of applications. A potential opportunity for AI tools is its capability of combine the information from the multiple image modalities (acquired at the baseline and during the patient follow-up) and clinical/genomic data to interpret the data or predict the outcome for a given task. This task could support the physicians in a multidisciplinary context producing a synergic support. In addition, AI tools might be used for building predictive models from the data, understanding of Radiomics, biological and genetics pathways to domain knowledge of a specific type of diseases and how they manifest in a given or multiple medical imaging modalities.
      Advanced AI efforts are focused on the complex tasks currently done by “human” experts/physicians. Surprisingly, despite a large portion of its daily routine is devoted to high-tech diagnostic imaging, there have still been fewer efforts of AI in diagnostic imaging [
      • Aktolun C.
      Artificial intelligence and radiomics in nuclear medicine: potentials and challenges.
      ] and this task might represent a future challenge [
      • Singh R.
      • Wu W.
      • Wang G.
      • Kalra M.K.
      Artificial intelligence in image reconstruction: the change is here.
      ].
      A potential role for AI-assisted diagnosis is the qualitative/quantitative image processing with extraction of radiomic features and their time-dependent changes. A strategic role for AI techniques would be the quality assurance of image device by reviewing the rawdata, detecting the motion artifacts and noise and correcting them by applying relevant correction tools without human supervision and intervention. The introduction of AI might change approval criteria and new quality control parameters and bring further challenges of validation standards.
      Another potential role for AI is the triage of requests by screening the available institutional digital archive (previous images, reports, tests) and distinguishing the patients with high probability of disease/recurrence. For follow-up purposes or in repetitive screening tests (such as mammography, chest X-rays, PETC/CT or CT imaging) AI could support physicians probably in a shorter time, with less interobserver variability and greater accuracy than “human” radiologists, more effort and time are required for implementing and validating AI tools. In addition, once the patients are scanned, the AI tools would be potentially coded to screen the scans and decide if additional scans in different positions/regions or at a different time point is needed. This feature would reduce the patient waiting time and physician workload. In this context, AI may potentially open the way to a better individualized patient care and precision medicine. AI would be of further identify the most recently published papers and guidelines that are relevant to the key findings/disease detected on the current scan aiding to prepare a most up-to-date customized patient report. This time-consuming focused literature search is already a part of routine clinical reporting for selected examinations in some academic departments.
      Dosimetry is another potential area of work for AI tools by performing precise segmentation of organs and lesions and calculations of absorbed dose using data obtained from AI improved three-dimensional imaging. This would be potentially useful for both diagnostic and therapeutic radionuclide procedures. Also, optimization of imaging equipment technology, criteria, and parameters through AI tools integrated to imaging devices can possibly bring the opportunity to reduce unnecessary patient absorbed dose (e.g. administered activity in nuclear medicine procedures). As another future direction is the potential applications of AI approaches in the radiotherapy workflow [
      • Meyer P.
      • Noblet V.
      • Mazzara C.
      • Lallement A.
      Survey on deep learning for radiotherapy.
      ]. A decision support tool can be used for patient consultation purposes. In the field of image processing, metal artifact reduction, synthetic CT from MRI and image quality improvements can be applied to the planning acquisition process (CT, MRI, PET). For target and structure segmentation, AI algorithms can be used for auto-detection/auto-segmentation of OARs and target volumes and for multi-modality image registration. Dose prediction tool and on-line adaptive radiotherapy application could be used to support the treatment planning process. During the treatment delivery, AI approaches could be used for the image-based motion managementand patient setup (e.g., object recognition, collision avoidance, intra-fraction motion prediction, surface guide radiotherapy).
      Once they are validated, AI tools has the potential to represent a standard in real-life clinical applications and to reduce repetitive, time-consuming, and monotonous tasks. This may even result in improved patient care and increased life quality of physicians in whom “burnout syndrome” due to heavy workload has been getting a serious problem [
      • Jalal S.
      • Parker W.
      • Ferguson D.
      • Nicolaou S.
      Exploring the role of artificial intelligence in an emergency and trauma radiology department.
      ].
      Finally, the implementation of AI programs containing sensitive health information leads to cyber security risk and their reduction/prevention represents another challenge of these tools in clinical practice [
      • Aktolun C.
      Artificial intelligence and radiomics in nuclear medicine: potentials and challenges.
      ].

      Conclusions

      Although the number of AI algorithms, studies and applications to medical imaging will increase rapidly in the coming years, the main obstacle to this development could be related to the lack of homogeneous training data. Multicentric studies will have an important role in this research area. The use of inter operable standards and homogeneous protocols will also be needed before such study can be performed. AI methods can provide useful models for quality assurance, personalized and predictive medicine. For this purpose, the contribute of clinicians and researchers in the interpretation of models and their application has a crucial role in the daily clinical practice.

      Funding

      This study was partially supported by Associazione Italiana per la Ricerca sul Cancro (AIRC) to LS (IG 2017 ID. 20809).

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

      1. Ranschaert E, Morozov S, Algra P. Artificial Intelligence in Medical Imaging Opportunities, Applications and Risks: Opportunities, Applications and Risks; 2019.

        • Lee J.G.
        • Jun S.
        • Cho Y.W.
        • Lee H.
        • Kim G.B.
        • Seo J.B.
        • et al.
        Deep learning in medical imaging: general overview.
        Korean J Radiol. 2017; 18: 570-584
        • Shi L.
        • Kusko R.
        • Wolfinger R.D.
        • Haibe-Kains B.
        • Fischer M.
        • Sansone S.A.
        • et al.
        The international MAQC Society launches to enhance reproducibility of high-throughput technologies.
        Nat Biotechnol. 2017; 35: 1127-1128
        • Mitchell T.M.
        Machine learning.
        McGraw-Hill, New York1997
        • Meyer P.
        • Noblet V.
        • Mazzara C.
        • Lallement A.
        Survey on deep learning for radiotherapy.
        Comput Biol Med. 2018; 98: 126-146
        • Sedik A.
        • Iliyasu A.M.
        • Abd El-Rahiem B.
        • Abdel Samea M.E.
        • Abdel-Raheem A.
        • Hammad M.
        • et al.
        Deploying machine and deep learning models for efficient data-augmented detection of COVID-19 infections.
        Viruses. 2020; 12
        • Min S.
        • Lee B.
        • Yoon S.
        Deep learning in bioinformatics.
        Brief Bioinf. 2017; 18: 851-869
        • Schmidhuber J.
        Deep learning in neural networks: an overview.
        Neural Netw. 2015; 61: 85-117
        • Motozawa N.
        • An G.
        • Takagi S.
        • Kitahata S.
        • Mandai M.
        • Hirami Y.
        • et al.
        Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes.
        Ophthalmol Ther. 2019; 8: 527-539
        • Lee C.S.
        • Nagy P.G.
        • Weaver S.J.
        • Newman-Toker D.E.
        Cognitive and system factors contributing to diagnostic errors in radiology.
        AJR Am J Roentgenol. 2013; 201: 611-617
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339b2535
        • Shahin M.A.
        • Maier H.R.
        • Jaska M.B.
        Data division for developing neural networks applied to geotechnical engineering.
        J Comput Civil Eng. 2004; 18: 105-114
        • Trivizakis E.
        • Ioannidis G.S.
        • Melissianos V.D.
        • Papadakis G.Z.
        • Tsatsakis A.
        • Spandidos D.A.
        • et al.
        A novel deep learning architecture outperforming 'off-the-shelf' transfer learning and feature-based methods in the automated assessment of mammographic breast density.
        Oncol Rep. 2019; 42: 2009-2015
        • Lamy J.B.
        • Sekar B.
        • Guezennec G.
        • Bouaud J.
        • Séroussi B.
        Explainable artificial intelligence for breast cancer: a visual case-based reasoning approach.
        Artif Intell Med. 2019; 94: 42-53
        • Choudhury N.
        • Begum S.
        A survey on case-based reasoning in medicine.
        Int J Adv Comput Sci Appl. 2016; 7
        • Damiani G.
        • Pinnarelli L.
        • Colosimo S.C.
        • Almiento R.
        • Sicuro L.
        • Galasso R.
        • et al.
        The effectiveness of computerized clinical guidelines in the process of care: a systematic review.
        BMC Health Serv Res. 2010; 10: 2
        • VoPham T.
        • Hart J.E.
        • Laden F.
        • Chiang Y.Y.
        Emerging trends in geospatial artificial intelligence (geoAI): potential applications for environmental epidemiology.
        Environ Health. 2018; 17: 40
        • Tschandl P.
        • Rosendahl C.
        • Kittler H.
        The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.
        Sci Data. 2018; 5180161
        • LeCun Y.
        • Bengio Y.
        • Hinton G.
        Deep learning.
        Nature. 2015; 521: 436-444
        • Hoseini F.
        • Shahbahrami A.
        • Bayat P.
        AdaptAhead optimization algorithm for learning deep CNN applied to MRI segmentation.
        J Digit Imaging. 2019; 32: 105-115
        • Li H.
        • Parikh N.A.
        • He L.
        A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes.
        Front Neurosci. 2018; 12: 491
        • Kiani A.
        • Uyumazturk B.
        • Rajpurkar P.
        • Wang A.
        • Gao R.
        • Jones E.
        • et al.
        Impact of a deep learning assistant on the histopathologic classification of liver cancer.
        NPJ Digit Med. 2020; 3: 23
        • Mobadersany P.
        • Yousefi S.
        • Amgad M.
        • Gutman D.A.
        • Barnholtz-Sloan J.S.
        • Velázquez Vega J.E.
        • et al.
        Predicting cancer outcomes from histology and genomics using convolutional networks.
        Proc Natl Acad Sci U S A. 2018; 115: E2970-E2979
        • Grossman R.L.
        • Heath A.P.
        • Ferretti V.
        • Varmus H.E.
        • Lowy D.R.
        • Kibbe W.A.
        • et al.
        Toward a shared vision for cancer genomic data.
        N Engl J Med. 2016; 375: 1109-1112
        • Ovchinnikova K.
        • Kovalev V.
        • Stuart L.
        • Alexandrov T.
        OffsampleAI: artificial intelligence approach to recognize off-sample mass spectrometry images.
        BMC Bioinf. 2020; 21: 129
        • Borkowski A.A.
        • Viswanadhan N.A.
        • Thomas L.B.
        • Guzman R.D.
        • Deland L.A.
        • Mastorides S.M.
        Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis.
        Fed Pract. 2020; 37: 398-404
      2. Deng J, Dong W, Socher R, Li L, Kai L, Li F-F, editors. ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition; 2009 20-25 June 2009.

      3. Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M, editors. Deep Learning Human Mind for Automated Visual Classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 21-26 July 2017.

        • Fares A.
        • Zhong S.H.
        • Jiang J.
        EEG-based image classification via a region-level stacked bi-directional deep learning framework.
        BMC Med Inf Decis Making. 2019; 19: 268
        • Huang Y.
        • Xu J.
        • Zhou Y.
        • Tong T.
        • Zhuang X.
        Diagnosis of Alzheimer's disease via multi-modality 3D convolutional neural network.
        Front Neurosci. 2019; 13: 509
        • Jin D.
        • Zhou B.
        • Han Y.
        • Ren J.
        • Han T.
        • Liu B.
        • et al.
        Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer's disease.
        Adv Sci (Weinh). 2020; 7: 2000675
        • Lawrimore J.
        • Doshi A.
        • Walker B.
        • Bloom K.
        AI-assisted forward modeling of biological structures.
        Front Cell Dev Biol. 2019; 7: 279
        • Kusumoto D.
        • Lachmann M.
        • Kunihiro T.
        • Yuasa S.
        • Kishino Y.
        • Kimura M.
        • et al.
        Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells.
        Stem Cell Rep. 2018; 10: 1687-1695
        • Fischer C.A.
        • Besora-Casals L.
        • Rolland S.G.
        • Haeussler S.
        • Singh K.
        • Duchen M.
        • et al.
        MitoSegNet: easy-to-use deep learning segmentation for analyzing mitochondrial morphology.
        iScience. 2020; 23 (101601)
        • Zhang Y.
        • Hamada M.
        DeepM6ASeq: prediction and characterization of m6A-containing sequences using deep learning.
        BMC Bioinf. 2018; 19: 524
        • Yamada K.D.
        • Kinoshita K.
        De novo profile generation based on sequence context specificity with the long short-term memory network.
        BMC Bioinf. 2018; 19: 272
        • Ho Thanh Lam L.
        • Le N.H.
        • Van Tuan L.
        • Tran Ban H.
        • Nguyen Khanh Hung T.
        • Nguyen N.T.K.
        • et al.
        Machine learning model for identifying antioxidant proteins using features calculated from primary sequences.
        Biology (Basel). 2020; 9
        • Yao K.
        • Rochman N.D.
        • Sun S.X.
        CTRL - a label-free artificial intelligence method for dynamic measurement of single-cell volume.
        J Cell Sci. 2020; 133
        • Smolander J.
        • Stupnikov A.
        • Glazko G.
        • Dehmer M.
        • Emmert-Streib F.
        Comparing biological information contained in mRNA and non-coding RNAs for classification of lung cancer patients.
        BMC Cancer. 2019; 19: 1176
        • Kwon J.M.
        • Kim K.H.
        • Jeon K.H.
        • Kim H.M.
        • Kim M.J.
        • Lim S.M.
        • et al.
        Development and validation of deep-learning algorithm for electrocardiography-based heart failure identification.
        Korean Circ J. 2019; 49: 629-639
        • Kwon J.M.
        • Kim K.H.
        • Jeon K.H.
        • Lee S.E.
        • Lee H.Y.
        • Cho H.J.
        • et al.
        Artificial intelligence algorithm for predicting mortality of patients with acute heart failure.
        PLoS ONE. 2019; 14e0219302
        • Kwon J.M.
        • Lee Y.
        • Lee Y.
        • Lee S.
        • Park J.
        An Algorithm based on deep learning for predicting in-hospital cardiac arrest.
        J Am Heart Assoc. 2018; 7
        • Diller G.P.
        • Babu-Narayan S.
        • Li W.
        • Radojevic J.
        • Kempny A.
        • Uebing A.
        • et al.
        Utility of machine learning algorithms in assessing patients with a systemic right ventricle.
        Eur Heart J Cardiovasc Imaging. 2019; 20: 925-931
        • Lin C.S.
        • Lin C.
        • Fang W.H.
        • Hsu C.J.
        • Chen S.J.
        • Huang K.H.
        • et al.
        A Deep-Learning Algorithm (ECG12Net) for detecting hypokalemia and hyperkalemia by electrocardiography: algorithm development.
        JMIR Med Inf. 2020; 8e15931
        • Attia Z.I.
        • Sugrue A.
        • Asirvatham S.J.
        • Ackerman M.J.
        • Kapa S.
        • Friedman P.A.
        • et al.
        Noninvasive assessment of dofetilide plasma concentration using a deep learning (neural network) analysis of the surface electrocardiogram: a proof of concept study.
        PLoS ONE. 2018; 13e0201059
        • Hart S.N.
        • Flotte W.
        • Norgan A.P.
        • Shah K.K.
        • Buchan Z.R.
        • Mounajjed T.
        • et al.
        Classification of melanocytic lesions in selected and whole-slide images via convolutional neural networks.
        J Pathol Inf. 2019; 10: 5
        • Hekler A.
        • Utikal J.S.
        • Enk A.H.
        • Hauschild A.
        • Weichenthal M.
        • Maron R.C.
        • et al.
        Superior skin cancer classification by the combination of human and artificial intelligence.
        Eur J Cancer. 2019; 120: 114-121
        • Walker B.N.
        • Rehg J.M.
        • Kalra A.
        • Winters R.M.
        • Drews P.
        • Dascalu J.
        • et al.
        Dermoscopy diagnosis of cancerous lesions utilizing dual deep learning algorithms via visual and audio (sonification) outputs: laboratory and prospective observational studies.
        EBioMedicine. 2019; 40: 176-183
        • Maron R.C.
        • Utikal J.S.
        • Hekler A.
        • Hauschild A.
        • Sattler E.
        • Sondermann W.
        • et al.
        Artificial intelligence and its effect on dermatologists' accuracy in dermoscopic melanoma image classification: web-based survey study.
        J Med Internet Res. 2020; 22e18091
        • Han S.S.
        • Park G.H.
        • Lim W.
        • Kim M.S.
        • Na J.I.
        • Park I.
        • et al.
        Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network.
        PLoS ONE. 2018; 13e0191493
        • Bizzego A.
        • Bussola N.
        • Chierici M.
        • Maggio V.
        • Francescatto M.
        • Cima L.
        • et al.
        Evaluating reproducibility of AI algorithms in digital pathology with DAPPER.
        PLoS Comput Biol. 2019; 15e1006269
        • Jones A.D.
        • Graff J.P.
        • Darrow M.
        • Borowsky A.
        • Olson K.A.
        • Gandour-Edwards R.
        • et al.
        Impact of pre-analytical variables on deep learning accuracy in histopathology.
        Histopathology. 2019; 75: 39-53
        • Ehteshami Bejnordi B.
        • Veta M.
        • Johannes van Diest P.
        • van Ginneken B.
        • Karssemeijer N.
        • Litjens G.
        • et al.
        Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.
        JAMA. 2017; 318: 2199-2210
        • Palatnik de Sousa I.
        • Maria Bernardes Rebuzzi Vellasco M.
        • Costa da Silva E.
        Local interpretable model-agnostic explanations for classification of lymph node metastases.
        Sensors (Basel). 2019; 19
        • Vandenberghe M.E.
        • Scott M.L.
        • Scorer P.W.
        • Söderberg M.
        • Balcerzak D.
        • Barker C.
        Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer.
        Sci Rep. 2017; 7: 45938
        • Khosravi P.
        • Kazemi E.
        • Zhan Q.
        • Malmsten J.E.
        • Toschi M.
        • Zisimopoulos P.
        • et al.
        Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization.
        NPJ Digit Med. 2019; 2: 21