- •Machine learning reconstruction of MRI data is becoming increasingly popular in research.
- •Many methods exist to perform machine learning reconstruction of MRI data.
- •The limited availability of publicly available training data sets, restricts current development and comparison of existing methods.
- •There is currently very limited clinical validation of MRI images reconstructed using machine learning.
1.1 The image reconstruction problem
1.2 Deep learning
2. Supervised machine learning
2.1 Image restoration methods
2.2 k-space methods
2.3 Direct mapping
2.4 Cross-domain methods
- Souza R.
- Lebel R.M.
- Hybrid F.R.A.
2.5 Unrolled optimization
where is a generic data consistency term, which ensures that the solution agrees with the observations , and is a generic regularization term which incorporates prior information. The definitions of and , together with the optimization strategy, determine the fundamental structure of the resulting neural network. Several approaches are outlined hereafter. A summary of the techniques described is presented in Table 1, which the reader is encouraged to use for reference.
|ADMM-Net||ADMM||(Conv), (implicit, proximal operator, piecewise linear function)|
|VarNet||GD||(Conv), (implicit, first order derivative, radial basis functions)|
|R-GANCS||PGD||(implicit, proximal operator, GAN).|
|HC-PGD||PGD||(implicit, proximal operator, CNN).|
|VS-Net||AMA||(implicit, proximal operator, CNN).|
|CRNN-MRI||AMA||(implicit, proximal operator, CRNN).|
|PDHG-CSNet||PDHG||(implicit, proximal operator, CNN).|
|CP-Net, PD-Net||PDHG||, (implicit, proximal operators, CNN’s)|
3. Unsupervised machine learning
4. Clinical implications
5. Current limitations
Declaration of Competing Interest
- SENSE: Sensitivity encoding for fast MRI.MRM. 1999; 42: 952-962
- Generalized autocalibrating partially parallel acquisitions (GRAPPA).MRM. 2002; 47: 1202-1210
- Compressed Sensing MRI.IEEE Signal Process Mag. 2008; 25: 72-82
- Convolutional networks for images, speech, and time series.MIT Press, The handbook of brain theory and neural networks1998: 255-258
- Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge.Magn Reson Med. 2020; 84: 3054-3070
- MR image reconstruction using deep learning: evaluation of network structure and loss functions.Quant Imaging Med Surg. 2019; 9: 1516-1527
Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for neural networks for image processing. arXiv preprint arXiv:151108861, 2015.
Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, p. 586–95.
- Accelerating magnetic resonance imaging via deep learning.in: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI): IEEE. 2016: 514-517
- Convolutional networks for biomedical image segmentation.in: International Conference on Medical image computing and computer-assisted intervention. Springer, 2015: 234-241
Lee D, Yoo J, Ye JC. Deep residual learning for compressed sensing MRI. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)2017. p. 15–8.
- Deep convolutional neural networks for accelerated dynamic magnetic resonance imaging.preprint. 2017;
- Spatio-temporal deep learning-based undersampling artefact reduction for 2D Radial Cine MRI with limited training data.IEEE Trans Med Imaging. 2020; 39: 703-717
- Real-time cardiovascular MR with spatio-temporal artifact suppression using deep learning–proof of concept in congenital heart disease.Magn Reson Med. 2019; 81: 1143-1156
Nath R, Callahan S, Singam N, Stoddard M, Amini AA. Accelerated Phase Contrast Magnetic Resonance Imaging via Deep Learning. 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020, p. 834–8.
- Generative adversarial networks.Adv Neural Inf Process Syst. 2014; 3
- DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction.IEEE Trans Med Imaging. 2018; 37: 1310-1321
- Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss.IEEE Trans Med Imaging. 2018; 37: 1488-1497
- Deep generative adversarial neural networks for compressive sensing MRI.IEEE Trans Med Imaging. 2019; 38: 167-179
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. p. 770–8.
- Image super-resolution using deep convolutional networks.IEEE Trans Pattern Anal Mach Intell. 2015; 38: 295-307
Cherukuri V, Guo T, Schiff SJ, Monga V. Deep MR image super-resolution using structural priors. 2018 25th IEEE International Conference on Image Processing (ICIP): IEEE; 2018. p. 410–4.
Pham C-H, Ducournau A, Fablet R, Rousseau F. Brain MRI super-resolution using deep 3D convolutional networks. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017): IEEE; 2017. p. 197–200.
- Deep learning single-frame and multiframe super-resolution for cardiac MRI.Radiology. 2020; 295: 552-561
Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D. Brain MRI super resolution using 3D deep densely connected neural networks. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018): IEEE; 2018. p. 739–42.
Du J, Wang L, Gholipour A, He Z, Jia Y. Accelerated super-resolution MR image reconstruction via a 3D densely connected deep convolutional neural network. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM): IEEE; 2018. p. 349–55.
- Rapid whole-heart CMR with single volume super-resolution.J Cardiovasc Magn Reson. 2020; 22: 56
- A review of the deep learning methods for medical images super resolution problems.IRBM. 2020;
Cheng JY, Mardani M, Alley MT, Pauly JM, Vasanawala SS. DeepSPIRiT: Generalized parallel imaging using deep convolutional neural networks. Annual Meeting of the International Society of Magnetic Resonance in Medicine, 2018.
- SPIRiT: Iterative self-consistent parallel imaging reconstruction from arbitrary k-space.Magn Reson Med. 2010; 64: 457-471
- A software channel compression technique for faster reconstruction with many channels.Magn Reson Imaging. 2008; 26: 133-141
Du T, Zhang H, Song HK, Fan Y. Adaptive convolutional neural networks for k-space data interpolation in fast magnetic resonance imaging. arXiv:200601385, 2020.
- A general framework for compressed sensing and parallel MRI using annihilating filter based low-rank Hankel matrix.IEEE Trans Comput Imaging. 2016; 2: 480-495
- k-Space Deep Learning for Accelerated MRI.IEEE Trans Med Imaging. 2019; 39: 377-386
Cha E, Kim EY, Ye JC. k-space deep learning for parallel mri: Application to time-resolved mr angiography. arXiv:180600806, 2018.
- Scan-specific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: Database-free deep learning for fast imaging.Magn Reson Med. 2019; 81: 439-453
- Accelerated coronary MRI with sRAKI: A database-free self-consistent neural network k-space reconstruction for arbitrary undersampling.PLoS ONE. 2020; 15e0229418
Zhang C, Moeller S, Weingärtner S, Uğurbil K, Akçakaya M. Accelerated MRI using residual RAKI: Scan-specific learning of reconstruction artifacts. Annual Meeting of the International Society of Magnetic Resonance in Medicine, 2019.
- Low-rank modeling of local k-space neighborhoods (LORAKS) for constrained MRI.IEEE Trans Med Imaging. 2013; 33: 668-681
Kim TH, Garg P, Haldar JP. LORAKI: Autocalibrated recurrent neural networks for autoregressive MRI reconstruction in k-space. arXiv:190409390, 2019.
- Image reconstruction by domain-transform manifold learning.Nature. 2018; 555: 487-492
Schlemper J, Oksuz I, Clough JR, Duan J, King AP, Schnabel JA, et al. dAUTOMAP: Decomposing AUTOMAP to achieve scalability and enhance performance. arXiv:190910995, 2019.
Eo T, Shin H, Kim T, Jun Y, Hwang D. Translation of 1d inverse fourier transform of k-space to an image based on deep learning for accelerating magnetic resonance imaging. International Conference on Medical Image Computing and Computer-Assisted Intervention: Springer; 2018. p. 241–9.
- Accelerating Cartesian MRI by domain-transform manifold learning in phase-encoding direction.Med Image Anal. 2020; 101689
- Springer International Publishing, Cham2018: 12-20 ETER-net: End to end MR image reconstruction using recurrent neural network.
- A hybrid frequency-domain/image-domain deep network for magnetic resonance image reconstruction.in: 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). 2019: 257-264
El-Rewaidy H, Fahmy AS, Pashakhanloo F, Cai X, Kucukseymen S, Csecs I, et al. Multi-domain convolutional neural network (MD-CNN) for radial reconstruction of dynamic cardiac MRI. Magnetic Resonance in Medicine. n/a.
- KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images.Magn Reson Med. 2018; 80: 2188-2201
- Dual domain, cascade of convolutional neural networks for magnetic resonance image reconstruction.in: Cardoso M.J. Aasa F. Ben G. Ender K. Ipek O. Gozde U. Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning. Proceedings of Machine Learning Research: PMLR. 2019: 437-446
- A Deep Cascade of convolutional neural networks for dynamic MR image reconstruction.IEEE Trans Med Imaging. 2018; 37: 491-503
- Dual-domain cascade of U-nets for multi-channel magnetic resonance image reconstruction.Magn Reson Imaging. 2020; 71: 140-153
- A dual-domain deep lattice network for rapid MRI reconstruction.Neurocomputing. 2020; 397: 94-107
Jethi AK, Murugesan B, Ram K, Sivaprakasam M. Dual-Encoder-Unet For Fast Mri Reconstruction. 2020 IEEE 17th International Symposium on Biomedical Imaging Workshops (ISBI Workshops)2020. p. 1–4.
- IKWI-net: A cross-domain convolutional neural network for undersampled magnetic resonance image reconstruction.Magn Reson Imaging. 2020; 73: 1-10
- Learning a variational network for reconstruction of accelerated MRI data.Magn Reson Med. 2018; 79: 3055-3071
- Dense recurrent neural networks for accelerated MRI: History-cognizant unrolling of optimization algorithms.IEEE J Sel Top Signal Process. 2020; 14: 1280-1291
Mardani M, Monajemi H, Papyan V, Vasanawala S, Donoho D, Pauly J. Recurrent generative adversarial networks for proximal learning and automated compressive image recovery. arXiv preprint arXiv:171110046, 2017.
Zhang J, Ghanem B. ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. p. 1828–37.
- MoDL: Model-based deep learning architecture for inverse problems.IEEE Trans Med Imaging. 2019; 38: 394-405
- Springer International Publishing, Cham2019: 713-722 VS-Net: Variable splitting network for accelerated parallel MRI reconstruction.
- Convolutional recurrent neural networks for dynamic MR image reconstruction.IEEE Trans Med Imaging. 2019; 38: 280-290
Yang Y, Sun J, Li H, Xu Z. ADMM-Net: A deep learning approach for compressive sensing MRI. arXiv:170506869, 2017.
- ADMM-CSNet: A deep learning approach for image compressive sensing.IEEE Trans Pattern Anal Mach Intell. 2020; 42: 521-538
- A deep unrolling network inspired by total variation for compressed sensing MRI.Digital Signal Process. 2020; 107102856
- Deep ADMM-net for compressive sensing MRI.Adv Neural Inf Process Syst. 2016; : 10-18
- Dynamic MRI using model-based deep learning and SToRM priors: MoDL-SToRM.Magn Reson Med. 2019; 82: 485-494
- Springer, 2019: 21-29 Model learning: Primal dual networks for fast MR imaging.
Cheng J, Wang H, Zhu Y, Liu Q, Zhang Q, Su T, et al. Model-based Deep Medical Imaging: the roadmap of generalizing iterative reconstruction model using deep learning. arXiv:190608143, 2019.
Adler J, Öktem O. Learned Primal-dual Reconstruction. arXiv:170706474, 2017.
Putzky P, Welling M. Recurrent inference machines for solving inverse problems. arXiv preprint arXiv:170604008, 2017.
- Recurrent inference machines for reconstructing heterogeneous MRI data.Med Image Anal. 2019; 53: 64-78
- Invert to learn to invert.Adv Neural Inf Process Syst. 2019; : 446-456
Cole EK, Pauly JM, Vasanawala SS, Ong F. Unsupervised MRI Reconstruction with Generative Adversarial Networks. arXiv preprint arXiv:200813065, 2020.
Tamir JI, Yu SX, Lustig M. Unsupervised Deep Basis Pursuit: Learning inverse problems without ground-truth data. arXiv preprint arXiv:191013110, 2019.
Lehtinen J, Munkberg J, Hasselgren J, Laine S, Karras T, Aittala M, et al. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:180304189, 2018.
- RARE: Image reconstruction using deep priors learned without groundtruth.IEEE J Sel Top Signal Process. 2020; 14: 1088-1099
- Super-resolution musculoskeletal MRI using deep learning.Magn Reson Med. 2018; 80: 2139-2154
Ravishankar S, Lahiri A, Blocker C, Fessler JA. Deep dictionary-transform learning for image reconstruction. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018. p. 1208–12.
- Data-driven learning of a union of sparsifying transforms model for blind compressed sensing.IEEE Trans Comput Imaging. 2016; 2: 294-309
- Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data.in: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI): IEEE. 2020: 921-925
- Compressed sensing using generative models.in: Doina P. Yee Whye T. Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research: PMLR. 2017: 537-546
- Inverse GANs for accelerated MRI reconstruction.SPIE. 2019;
- CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions.Sci Rep. 2020; 10: 13710
Haji-Valizadeh H, Shen D, Avery RJ, Serhal AM, Schiffers FA, Katsaggelos AK, et al. Rapid reconstruction of four-dimensional MR angiography of the thoracic aorta using a convolutional neural network. Radiology: Cardiothoracic Imaging. 2020;2:e190205.
- A multi-scale variational neural network for accelerating motion-compensated whole-heart 3D coronary MR angiography.Magn Reson Imaging. 2020; 70: 155-167
- Deep complex convolutional network for fast reconstruction of 3D late gadolinium enhancement cardiac MRI.NMR Biomed. 2020; 33e4312
- Deep variational network for rapid 4D flow MRI reconstruction.Nature Machine Intell. 2020; 2: 228-235
- Deep residual learning for accelerated MRI using magnitude and phase networks.IEEE Trans Biomed Eng. 2018; 65: 1985-1995
Virtue P, Yu SX, Lustig M. Better than real: Complex-valued neural nets for MRI fingerprinting. 2017 IEEE International Conference on Image Processing (ICIP), 2017. p. 3953–7.
Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos J, et al. Deep Complex Networks. arXiv:170509792.
Cole E, Cheng J, Pauly J, Vasanawala S. Analysis of deep complex-valued convolutional neural networks for MRI reconstruction.
- A parallel MR imaging method using multilayer perceptron.Med Phys. 2017; 44: 6209-6224
Knoll F, Zbontar J, Sriram A, Muckley MJ, Bruno M, Defazio A, et al. fastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning. radiology: Artificial intelligence. 2020;2:e190007.
- An open, multi-vendor, multi-field-strength brain MR dataset and analysis of publicly available skull stripping methods agreement.NeuroImage. 2018; 170: 482-494
- UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age.PloS Med. 2015; 12e1001779
- The WU-Minn human connectome project: An overview.NeuroImage. 2013; 80: 62-79
- Online database of clinical MR and ultrasound images of brain tumors.Med Phys. 2012; 39: 3253-3261
LaMontagne PJ, Benzinger TL, Morris JC, Keefe S, Hornbeck R, Xiong C, et al. OASIS-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease. medRxiv. 2019:2019.12.13.19014902.
- Benchmarking MRI reconstruction neural networks on large public datasets.Appl Sci. 2020; 10: 1816
- Gadgetron: An open source framework for medical image reconstruction.Magn Reson Med. 2013; 69: 1768-1776