Advertisement
Original paper| Volume 107, 102535, March 2023

Download started.

Ok

Deep image and feature prior algorithm based on U-ConformerNet structure

  • Zhengming Yi
    Affiliations
    The State Key Laboratory of Refractories and Metallurgy, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China

    Key Laboratory for Ferrous Metallurgy and Resources Utilization of Metallurgy and Resources Utilization of Education, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China
    Search for articles by this author
  • Junjie Wang
    Affiliations
    The State Key Laboratory of Refractories and Metallurgy, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China

    Key Laboratory for Ferrous Metallurgy and Resources Utilization of Metallurgy and Resources Utilization of Education, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China
    Search for articles by this author
  • Mingjie Li
    Correspondence
    Corresponding author at: The State Key Laboratory of Refractories and Metallurgy, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China.
    Affiliations
    The State Key Laboratory of Refractories and Metallurgy, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China

    Key Laboratory for Ferrous Metallurgy and Resources Utilization of Metallurgy and Resources Utilization of Education, Wuhan University of Science and Technology, Wuhan 430081, Hubei, China
    Search for articles by this author
Published:February 08, 2023DOI:https://doi.org/10.1016/j.ejmp.2023.102535

      Highlights

      • The U-ConformerNet structure is used to exploit the potential of DIP algorithm.
      • The feature distance regularization is used to enhance image quality and detail.
      • The algorithm can be freely switched between supervised and unsupervised modes.
      • The regularization based on image feature is better than traditional TV constraint.

      Abstract

      Purpose

      The reconstruction performance of the deep image prior (DIP) approach is limited by the conventional convolutional layer structure and it is difficult to enhance its potential. In order to improve the quality of image reconstruction and suppress artifacts, we propose a DIP algorithm with better performance, and verify its superiority in the latest case.

      Methods

      We construct a new U-ConformerNet structure as the DIP algorithm's network, replacing the traditional convolutional layer-based U-net structure, and introduce the 'lpips' deep network based feature distance regularization method. Our algorithm can switch between supervised and unsupervised modes at will to meet different needs.

      Results

      The reconstruction was performed on the low dose CT dataset (LoDoPaB). Our algorithm attained a PSNR of more than 35 dB under unsupervised conditions, and the PSNR under the supervised condition is greater than 36 dB. Both of which are better than the performance of the DIP-TV. Furthermore, the accuracy of this method is positively connected with the quality of the a priori image with the help of deep networks. In terms of noise eradication and artifact suppression, the DIP algorithm with U-ConformerNet structure outperforms the standard DIP method based on convolutional structure.

      Conclusions

      It is known by experimental verification that, in unsupervised mode, the algorithm improves the output PSNR by at least 2–3 dB when compared to the DIP-TV algorithm (proposed in 2020). In supervised mode, our algorithm approaches that of the state-of-the-art end-to-end deep learning algorithms.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Physica Medica: European Journal of Medical Physics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Beister M.
        • Kolditz D.
        • Kalender W.A.
        Iterative reconstruction methods in X-ray CT.
        Phys Med. 2012; 28: 94-108https://doi.org/10.1016/j.ejmp.2012.01.003
      1. Jin X, Li L, Chen Z, Zhang L, Xing Y, Anisotropic total variation for limited-angle CT reconstruction. In IEEE Nuclear Science Symposuim & Medical Imaging Conference. 2010. IEEE. 10.1109/NSSMIC.2010.5874180.

        • Ben Y.
        • Cardoen H.B.
        • Hamarneh G.
        Deep learning for biomedical image reconstruction: a survey.
        Artif Intell Rev. 2021; 54: 215-251https://doi.org/10.1007/s10462-020-09861-2
        • Després P.
        • Jia X.
        A review of GPU-based medical image reconstruction.
        Phys Med. 2017; 42: 76-92https://doi.org/10.1016/j.ejmp.2017.07.024
        • Singh R.
        • Wu W.
        • Wang G.
        • Kalra M.K.
        Artificial intelligence in image reconstruction: The change is here.
        Phys Med. 2020; 79: 113-125https://doi.org/10.1016/j.ejmp.2020.11.012
        • Wang T.
        • Lei Y.
        • Fu Y.
        • Curran W.J.
        • Liu T.
        • Nye J.A.
        • et al.
        Machine learning in quantitative PET: A review of attenuation correction and low-count image reconstruction methods.
        Phys Med. 2020; 76: 294-306https://doi.org/10.1016/j.ejmp.2020.07.028
        • Castiglioni I.
        • Rundo L.
        • Codari M.
        • Leo G.D.
        • Salvatore C.
        • Interlenghi M.
        • et al.
        AI applications to medical images: From machine learning to deep learning.
        Phys Med. 2021; 83: 9-24https://doi.org/10.1016/j.ejmp.2021.02.006
        • Manco L.
        • Maffei N.
        • Strolin S.
        • Vichi S.
        • Bottazzi L.
        • Strigari L.
        Basic of machine learning and deep learning in imaging for medical physicists.
        Phys Med. 2021; 83: 194-205https://doi.org/10.1016/j.ejmp.2021.03.026
        • Barragán-Montero A.
        • Umair J.
        • Valdés G.
        • Nguyen D.
        • Desbordes P.
        • Macq B.
        • et al.
        Artificial intelligence and machine learning for medical imaging: A technology review.
        Phys Med. 2021; 83: 242-256https://doi.org/10.1016/j.ejmp.2021.04.016
        • Balagurunathan Y.
        • Ross M.
        • Issam E.N.
        Requirements and reliability of AI in the medical context.
        Phys Med. 2021; 83: 72-78https://doi.org/10.1016/j.ejmp.2021.02.024
        • Saw S.N.
        • Kwan H.N.
        Current challenges of implementing artificial intelligence in medical imaging.
        Phys Med. 2022; 100: 12-17https://doi.org/10.1016/j.ejmp.2022.06.003
        • Zhang Q.
        • Hu Z.
        • Jiang C.
        • Zheng H.
        • Ge Y.
        • Liang D.
        Artifact removal using a hybrid-domain convolutional neural network for limited-angle computed tomography imaging.
        Phys Med Biol. 2020; 65155010https://doi.org/10.1088/1361-6560/ab9066
        • You C.
        • Yang Q.
        • Shan H.
        • Gjesteby L.
        • Li G.
        • Ju S.
        • et al.
        Structurally-Sensitive Multi-Scale Deep Neural Network for Low-Dose CT Denoising.
        IEEE Access. 2018; 6: 41839-41855https://doi.org/10.1109/ACCESS.2018.2858196
        • Tanimoto R.
        • Higuchi K.
        • Ishiguro T.
        • Mori K.
        • Kojo K.
        • Kojima T.
        • et al.
        Segmentation of renal tumors in CT images by 3D U-Net preserving rotational symmetry in axial slices.
        Opt Continuum. 2022; 1: 297-305https://doi.org/10.1364/OPTCON.451024
        • Mameli F.
        • Bertini M.
        • Galteri L.
        • Bimbo A.D.
        Image and video restoration and compression artefact removal using a NoGAN approach.
        in: Proceedings of the 28th ACM International Conference on Multimedia. 2020https://doi.org/10.1145/3394171.3414451
        • Racine D.
        • Becce F.
        • Viry A.
        • Monnin P.
        • Thomsen B.
        • Verdun F.R.
        • et al.
        Task-based characterization of a deep learning image reconstruction and comparison with filtered back-projection and a partial model-based iterative reconstruction in abdominal CT: A phantom study.
        Phys Med. 2020; 76: 28-37https://doi.org/10.1016/j.ejmp.2020.06.004
        • Adler J.
        and Öktem O, Solving ill-posed inverse problems using iterative deep neural networks.
        Inverse Prob. 2017; 33124007https://doi.org/10.1088/1361-6420/aa9581
        • Würfl T.
        • Hoffmann M.
        • Christlein V.
        • Breininger K.
        • Huang Y.
        • Unberath M.
        • et al.
        Deep Learning Computed Tomography: Learning Projection-Domain Weights From Image Domain in Limited Angle Problems.
        IEEE Trans Med Imaging. 2018; 37: 1454-1463https://doi.org/10.1109/TMI.2018.2833499
        • Wang J.
        • Zeng L.
        • Wang C.
        • Guo Y.
        ADMM-based deep reconstruction for limited-angle CT.
        Phys Med Biol. 2019; 64115011https://doi.org/10.1088/1361-6560/ab1aba
        • Du W.
        • Chen H.
        • Yang H.
        • Zhang Y.
        Disentangled generative adversarial network for low-dose CT.
        EURASIP J Adv Signal Process. 2021; 2021: 34https://doi.org/10.1186/s13634-021-00749-z
        • Chen H.
        • Zhang Y.
        • Chen Y.
        • Zhang J.
        • Zhang W.
        • Sun H.
        • et al.
        LEARN: Learned Experts’ Assessment-Based Reconstruction Network for Sparse-Data CT.
        IEEE Trans Med Imaging. 2018; 37: 1333-1347https://doi.org/10.1109/TMI.2018.2805692
        • Adler J.
        • Öktem O.
        Learned Primal-Dual Reconstruction.
        IEEE Trans Med Imaging. 2018; 37: 1322-1332https://doi.org/10.1109/TMI.2018.2799231
      2. Ronneberger O, Fischer P, and Brox T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. 2015. Springer. 10.1007/978-3-319-24574-4_28.

        • DA Ulyanov V.
        • Lempitsky V.
        Deep image prior.
        in: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2018: 9446-9454
        • Baguer D.O.
        • Leuschner J.
        • Schmidt M.
        Computed tomography reconstruction using deep image prior and learned reconstruction methods.
        Inverse Prob. 2020; 36094004https://doi.org/10.1088/1361-6420/aba415
      3. Chakrabarty P. and Maji S, The spectral bias of the deep image prior. arXiv preprint arXiv:1912.08905; 2019. doi:10.48550/arXiv.1912.08905.

        • Esser P.
        • Rombach R.
        • Ommer B.
        Taming transformers for high-resolution image synthesis.
        in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 12873-12883
        • Shi C.
        • Xiao Y.
        • Chen Z.
        Dual-domain sparse-view CT reconstruction with Transformers.
        Phys Med. 2022; 101: 1-7https://doi.org/10.1016/j.ejmp.2022.07.001
        • Han Y.
        • Ye J.C.
        Framing U-Net via Deep Convolutional Framelets: Application to Sparse-View CT.
        IEEE Trans Med Imaging. 2018; 37: 1418-1429https://doi.org/10.1109/TMI.2018.2823768
        • Peng Z.
        • Huang W.
        • Gu S.
        • Xie L.
        • Wang Y.
        • Jiao J.
        • et al.
        Conformer: Local Features Coupling Global Representations for Visual Recognition.
        in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021: 367-376
        • Zhu B.
        • Liu J.Z.
        • Cauley S.F.
        • Rosen B.R.
        • Rosen M.S.
        Image reconstruction by domain-transform manifold learning.
        Nature. 2018; 555: 487-492https://doi.org/10.1038/nature25988
        • Shen C.
        • Gonzalez Y.
        • Chen L.
        • Jiang S.B.
        • Jia X.
        Intelligent Parameter Tuning in Optimization-Based Iterative CT Reconstruction via Deep Reinforcement Learning.
        IEEE Trans Med Imaging. 2018; 37: 1430-1439https://doi.org/10.1109/TMI.2018.2823679
      4. Dranoshchuk AD, and Veselov AI. About perceptual quality estimation for image compression. 2019 Wave Electronics and its Application in Information and Telecommunication Systems (WECONF). IEEE, 2019. Doi: 10.1109/WECONF.2019.8840116.

        • Swinehart D.F.
        The Beer-Lambert Law.
        J Chem Educ. 1962; 39: 333https://doi.org/10.1021/ed039p333
        • Niu S.
        • Gao Y.
        • Bian Z.
        • Huang J.
        • Chen W.
        • Yu G.
        • et al.
        Sparse-view x-ray CT reconstruction via total generalized variation regularization.
        Phys Med Biol. 2014; 59: 2997-3017https://doi.org/10.1088/0031-9155/59/12/2997
        • Zhang R.
        • Isola P.
        • Efros A.A.
        • Shechtman E.
        • Wang O.
        The unreasonable effectiveness of deep features as a perceptual metric.
        in: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2018: 586-595
        • Leuschner J.
        • Schmidt M.
        • Baguer D.O.
        • Maass P.
        LoDoPaB-CT, a benchmark dataset for low-dose computed tomography reconstruction.
        Sci Data. 2021; 8: 109https://doi.org/10.1038/s41597-021-00893-z
        • Jin K.H.
        • McCann M.T.
        • Froustey E.
        • Unser M.
        Deep Convolutional Neural Network for Inverse Problems in Imaging.
        IEEE Trans Image Process. 2017; 26: 4509-4522https://doi.org/10.1109/TIP.2017.2713099