Highlights
- •Different segmentation pipelines may provide inconsistent quantification of brain structures.
- •The intra- and inter-method agreement between two popular segmentation software packages SPM12 and FreeSurfer v6.0.
- •SPM provides more consistent results both in the intra- and the inter-method agreement evaluation.
- •There are consistent biases in the estimates of gray matter and white matter between SPM and FreeSurfer.
- •To cross-validate the findings of each study against different segmentation methods before interpreting of the results.
Abstract
Purpose
The lack of inter-method agreement can produce inconsistent results in neuroimaging
studies. We evaluated the intra-method repeatability and the inter-method reproducibility
of two widely-used automatic segmentation methods for brain MRI: the FreeSurfer (FS)
and the Statistical Parametric Mapping (SPM) software packages.
Methods
We segmented the gray matter (GM), the white matter (WM) and subcortical structures
in test-retest MRI data of healthy volunteers from Kirby-21 and OASIS datasets. We
used Pearson’s correlation (r), Bland-Altman plot and Dice index to study intra-method
repeatability and inter-method reproducibility. In order to test whether different
processing methods affect the results of a neuroimaging-based group study, we carried
out a statistical comparison between male and female volume measures.
Results
A high correlation was found between test-retest volume measures for both SPM (r in
the 0.98–0.99 range) and FS (r in the 0.95–0.99 range). A non-null bias between test-retest
FS volumes was detected for GM and WM in the OASIS dataset. The inter-method reproducibility
analysis measured volume correlation values in the 0.72–0.98 range and the overlap
between the segmented structures assessed by the Dice index was in the 0.76–0.83 range.
SPM systematically provided significantly greater GM volumes and lower WM and subcortical
volumes with respect to FS. In the male vs. female brain volume comparisons, inconsistencies
arose for the OASIS dataset, where the gender-related differences appear subtler with
respect to the Kirby dataset.
Conclusions
The inter-method reproducibility should be evaluated before interpreting the results
of neuroimaging studies.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Physica Medica: European Journal of Medical PhysicsAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Magnetic resonance imaging deformation-based segmentation and temporal lobe epilepsy.J Digit Imaging. 2000; 13: 217-218https://doi.org/10.1053/jdim.2000.6897
- Segmentation, feature extraction, and multiclass brain tumor classification.J Digit Imaging. 2013; 26: 1141-1150https://doi.org/10.1007/s10278-013-9600-0
- Comparison and evaluation of segmentation techniques for brain mri using Gold Standard. Indian.J Sci Technol. 2016; 9https://doi.org/10.17485/ijst/2016/v9i46/106495
- Reliability of brain volume measurements: a test-retest dataset.Sci Data. 2014; 1: 1-9https://doi.org/10.1038/sdata.2014.37
- The reproducibility and sensitivity of brain tissue volume measurements derived from an SPM-based segmentation methodology.J Magn Reson Imaging. 2002; 15: 259-267https://doi.org/10.1002/jmri.10064
- Scan-rescan reliability of subcortical brain volumes derived from automated segmentation.Hum Brain Mapp. 2010; 00: 1751-1762https://doi.org/10.1002/hbm.20973
- For the Alzheimer’s disease neuroimaging initiative. Comparison of automated brain volume measures obtained with neuroQuant® and FreeSurfer.J Neuroimaging. 2015; 25: 721-727https://doi.org/10.1111/jon.12229
- Inter-method discrepancies in brain volume estimation may drive inconsistent findings in autism.Front Neurosci. 2016;10.; https://doi.org/10.3389/fnins.2016.00439
- Comparing manual and automatic segmentation of hippocampal volumes: reliability and validity issues in younger and older brains.Hum Brain Mapp. 2014; 2914: 4236-4248https://doi.org/10.1002/hbm.22473
- Quantitative comparison of SPM, FSL, and brainsuite for brain mr image segmentation.J Biomed Phys Eng. 2014; 4: 13-26
- Are there any gender differences in the hippocampus volume after head-size correction? A volumetric and voxel-based morphometric study.Neurosci Lett. 2014; 570: 119-123https://doi.org/10.1016/j.neulet.2014.04.013
- SIENA-XL for improving the assessment of gray and white matter volume changes on brain MRI.Hum Brain Mapp. 2018; https://doi.org/10.1002/hbm.23828
- Comparison of accuracy between FSL’s FIRST and Freesurfer for caudate nucleus and putamen segmentation.Sci Rep. 2017; 7: 1-9https://doi.org/10.1038/s41598-017-02584-5
- Validation of hippocampal volumes measured using a manual method and two automated methods (FreeSurfer and IBASPM) in chronic major depressive disorder.Neuroradiology. 2008; 50: 569-581https://doi.org/10.1007/s00234-008-0383-9
- MRI-derived measurements of human subcortical, ventricular and intracranial brain volumes: reliability effects of scan sessions, acquisition sequences, data analyses, scanner upgrade, scanner vendors and field strengths.Neuroimage. 2009; 46: 177-192
- Head size, age and gender adjustment in MRI studies: a necessary nuisance?.Neuroimage. 2010; 53: 1244-1255https://doi.org/10.1016/j.neuroimage.2010.06.025
Neuroimaging B members & collaborations of the WCFH. SPM, Statistical Parametric Mapping n.d. Available at: https://www.fil.ion.ucl.ac.uk/spm.
- SPM12 manual the FIL methods group (and honorary members).Funct Imaging Lab. 2013; : 475-481https://doi.org/10.1111/j.1365-294X.2006.02813.x
Imaging L for CNAAMC for BBF. FreeSurfer n.d. https://surfer.nmr.mgh.harvard.edu.
- FreeSurfer.Neuroimage. 2012; https://doi.org/10.1016/j.neuroimage.2012.01.021
- Multi-parametric neuroimaging reproducibility: a 3-T resource study.Neuroimage. 2011; 54: 2854-2866https://doi.org/10.1016/j.neuroimage.2010.11.047
- Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults.J Cogn Neurosci. 2007; 19: 1498-1507https://doi.org/10.1162/jocn.2007.19.9.1498
NITRC. NeuroImaging Tools & Resources Collaboratory n.d. Available at: https://www.nitrc.org/projects/multimodal.
NITRC. NeuroImaging Tools & Resources Collaboratory. n.d. Available at: https://www.nitrc.org/projects/oasis/.
- 101 labeled brain images and a consistent human cortical labeling protocol.Front Neurosci. 2012; 6: 1-12https://doi.org/10.3389/fnins.2012.00171
- The case for using the repeatability coefficient when calculating test-retest reliability.PLoS ONE. 2013; 8: 1-7https://doi.org/10.1371/journal.pone.0073990
- Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain.Neuron. 2002; 33: 341-355https://doi.org/10.1016/S0896-6273(02)00569-X
- A comparison of automated segmentation and manual tracing for quantifying hippocampal and amygdala volumes.Neuroimage. 2009; 45: 855-866https://doi.org/10.1016/j.neuroimage.2008.12.033
- Unified segmentation.Neuroimage. 2005; 26: 839-851https://doi.org/10.1016/j.neuroimage.2005.02.018
- Using the Bland-Altman method to measure agreement with repeated measures.Br J Anaesth. 2007; 99: 309-311https://doi.org/10.1093/bja/aem214
- Gender and age differences in normal adult human brain: voxel-based morphometric study.Hum Brain Mapp. 2011; 32: 1050-1058https://doi.org/10.1002/hbm.21088
- Sex differences in the adult human brain: evidence from 5216 UK biobank participants.Cereb Cortex. 2018; 28: 2959-2975https://doi.org/10.1093/cercor/bhy109
- A meta-analysis of sex differences in human brain structure.Neurosci Biobehav Rev. 2014; 39: 34-50https://doi.org/10.1016/j.neubiorev.2013.12.004
Cohen J. Statistical power analysis for the behavioral sciences, second edition. 1988. doi:10.1234/12345678.
- Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool.BMC Med Imaging. 2015; 15https://doi.org/10.1186/s12880-015-0068-x
- Reproducibility and bias in healthy brain segmentation: comparison of two popular neuroimaging platforms.Front Neurosci. 2016; 10: 1-8https://doi.org/10.3389/fnins.2016.00503
- Robustness of automated methods for brain volume measurements across different MRI field strengths.PLoS ONE. 2016; https://doi.org/10.1371/journal.pone.0165719
- Cortical thickness estimations of FreeSurfer and the CAT12 toolbox in patients with Alzheimer’s disease and healthy controls.J Neuroimaging. 2018; 28: 515-523https://doi.org/10.1111/jon.12521
- Automatic 3d intersubject registration of mr volumetric data in standardized talairach space.J Comput Assist Tomogr. 1994; https://doi.org/10.1097/00004728-199403000-00005
- DeepNAT: deep convolutional neural network for segmenting neuroanatomy.Neuroimage. 2018; https://doi.org/10.1016/j.neuroimage.2017.02.035
- VoxResNet: deep voxelwise residual networks for brain segmentation from 3D MR images.Neuroimage. 2018; https://doi.org/10.1016/j.neuroimage.2017.04.041
Article info
Publication history
Published online: August 05, 2019
Accepted:
July 19,
2019
Received in revised form:
June 12,
2019
Received:
March 19,
2019
Identification
Copyright
© 2019 Associazione Italiana di Fisica Medica. Published by Elsevier Ltd. All rights reserved.
