Deep Convolutional Neural Networks For 3D Image Analysis

Medical image analysis is an interdisciplinary field that integrates medical imaging,mathematical modeling,machine learning,statistical analysis and other disciplines,aiming to extract physiological and pathological information as well as medical knowledge through medical image processing and provides the basis for medical research,diagnosis,and treatment.Convolutional neural networks have been adopted in medical image analysis recently.However,medical images are mostly 3D data and have the problems such as high dimension,limited samples,and complicated physiological structure,so it is not the best choice to apply classical convolutional neural networks to medical images directly.In this article,we review the development history of convolutional neural networks and their applications in medical image analysis.We take the characteristics of medical images into account and develop medical image analysis methods based on slice-based 2D images,3D voxel images,and 3D surface images.We conduct in-depth research on 3D medical image analysis methods and apply them to COVID-19 detection,gender classification,autism diagnosis,model visualization,and other aspects.The specific work and main achievements are as follows:(1)A dual-branch combination network based on CT slices has been developed for COVID-19 diagnosis,which can simultaneously achieve accurate classification and segmentation of lesions.We develop a lesion attention module with the attention mechanism to integrate the intermediate segmentation results,thus making the classification branch more focused on the lesion area.We collect a large dataset of 1202 subjects from 10 hospitals in China and tested the proposed model.The results show that the classification accuracy of the dual-branch combination network is 96.74%,outperforming other models.With higher sensitivity,our model has significant advantages in detecting subtle lesions,which is helpful for the screening of early COVID-19 patients.We have developed an online diagnostic platform for COVID-19 CT images based on a dual-branch combination network,which is already available for the public.(2)A hybrid semi-supervised model for COVID-19 screening and segmentation with small samples is proposed.Two new semi-supervised approaches are proposed.Firstly,considering the high similarity between adjacent slices of CT images,a sliceconsistent regularization approach is proposed to find smooth manifolds using unlabeled data.In addition,combined with the structural characteristic of the dual-branch model,we propose a dual-branch consistent pseudo-label method,which improves the reliability of pseudo-labels.We conduct tests on both the collected and public datasets,and the results demonstrat that our semi-supervised dual-branch combination network has a significant performance improvement over the supervised model for COVID-19 classification and lesion segmentation.For the training with small samples,the semisupervised model can still achieve high accuracy,which is significantly higher than the supervised method.Our semi-supervised model has great application potential in the field of medical image analysis with few labeled samples.(3)A new 3D convolutional model named Dense-CAM is proposed to solve the low resolution of visualization results in 3D medical images.The traditional visualization methods,class activation mapping,can only visualize the features of the last layer and generate low-resolution activation maps,which is difficult to meet the requirements of the high-precision interpretability of medical images.Our model combines features from all layers of the model together through skip connections and achieves the visualization of the whole network.The model integrates multi-scale features from different layers with different resolutions to achieve 3D convolutional network visualization with high accuracy and high stability,which is significant for medical image research and clinical treatment.We test the gender classification using a large brain imaging dataset containing 6008 cases,and the classification results reach an accuracy of 92.93%.Meanwhile,Dense-CAM is used to identify the most different brain regions in sex classification.(4)We propose a novel framework of deep transfer learning for brain images,which solves the problem that transfer learning is difficult to be applied to 3D brain images.The proposed framework improves the accuracy of small sample brain image classification effectively.We transform the 3D cerebral cortex into the 3D surface and then map the cortical surface into the 2D plane image using optimal transport mapping.Using 2D deep models pre-trained on Image Net and fine-tuned with mapped images,we improve classification performance on small samples of brain images through transfer learning.Furthermore,we propose a two-stage transfer learning strategy to improve the classification performance of autism spectrum disorder by using gender classification as an intermediate task.The two-stage transfer learning gains more performance improvement than direct learning from Image Net.Moreover,for more complex 4D f MRI data,the amplitude of low-frequency fluctuation is introduced to compress the time dimension of the f MRI data.Then the transfer learning framework for 3D data is adopted,thus realizing the analysis of 4D f MRI data with 2D models.The proposed framework bridge the 2D pre-trained models and 3D or 4D brain images,providing a new light for high-dimensional medical image analysis.