Research On Real-time And Multi-modal Slice-to-volume Brain Image Registration Methods Based On Deep Learning

Slice-to-volume registration,which belongs to 2D/3D image registration,is the key technology in brain clinical application areas such as detection of brain tumor change,brain radiotherapy guidance,and brain multi-modal image fusion,etc.It has important research significance and high medical value.However,there are still many challenges for clinical slice-to-volume registration applications.First,to provide more comprehensive pathological tissue information for doctors,morphological structures and pathological information obtained by different imaging techniques should be registered and further integrated.Furthermore,there are no landmarks for many brain clinical applications and registration can only be implemented through medical imaging technologies.Besides,the registration process needs to be conducted in real-time for surgical navigation and radiotherapy.At last but not least,different from 2D image alignment,slice-to-volume registration has more degrees of freedom in spatial transformation,resulting in a more complex spatial relationship.The above challenges heavily aggravate the difficulties of real-time and multi-modal slice-to-volume brain image registration.Nowadays,traditional slice-to-volume medical image registration methods mainly rely on image intensity or extracted features,and the registration can be further implemented through similarity measures.Nevertheless,these methods need to traverse the slices at each position and orientation in the 3D volume,which is a time-consuming process.Moreover,they are easy to be trapped into the local minimum,resulting in misalignment.Furthermore,a deep learning strategy is proposed to achieve slice-to-volume registration for volume reconstruction.The regression-based neural networks can be trained off-line and the fetal brain MRI volume can be reconstructed in real-time in the application because only forward computation is needed.Unfortunately,this method is only suitable for mono-modal registration and the drawbacks of huge training data,complex network structures,and long training time limit its application in the clinical practice.In addition,detection of brain tumor change and brain radiotherapy still need to be handled.To address the above disadvantages,the motivation of this dissertation is to investigate more efficient,accurate,and robust slice-to-volume registration methods based on the deep learning strategy,which can be suitable for various real-time and multi-modal brain image registration scenarios.First,a real-time slice-to-volume brain image registration method based on the multi-label classification neural network is proposed.By introducing the idea of classification,the slice-to-volume registration can be converted into a multi-label classification problem.This method establishes a one-to-one mapping between slices and their related spatial transformations through convolutional neural networks to solve the multi-label classification problem.Moreover,silhouettes are utilized to unify the multi-modal images into the same metric,which can transform multi-modal image registration into mono-modal one.Besides,the number of training samples can be effectively decreased by reducing the d imension of the parameter space.Comparative experiments with regression-based neural network methods verify the feasibility and superiority of this method in detection of brain tumor change and radiotherapy applications.Furthermore,to address the multi-modal problem and extend the applicability of deep learning-based slice-to-volume registration methods,an image representation method based on self-learning and two-branch neural networks is proposed.A two-branch neural network is designed with weight sharing and different multi-modal brain image pairs can be simultaneously fed into two shared-weight channels.The network can be trained by making the paired outputs similar and retaining the edge information of the originals(i.e.,the self-learning strategy).In this way,the common features of brain multi-modal images are represented without strong target constrains.Different from multi-modal image translation methods,in which a network can only realize the translation between two modalities,our proposed method can represent several modal brain images through one network.Moreover,experiments demonstrate that more accurate and robust registration can be achieved when it is applied to deep learning-based slice-to-volume registration methods.At last,the shortcomings of huge training data storage,redundancy of the network structure,and a long time of network training for the deep learning-based methods limit the practical clinical application of slice-to-volume registration based on deep learning.To further extend the applicability of deep learning-based slice-to-volume registration methods,a fast slice-to-volume registration algorithm based on multi-channel one-dimensional convolutional neural networks is proposed.A multi-angle projection method is first investigated to transform the 2D image into multi-channel one-dimensional data,which effectively reduces the storage of training data.Meanwhile,a multi-channel one-dimensional convolutional neural network is established to greatly reduce the network complexity and the number of parameters,resulting in network compression.Experiments can demonstrate that this method can greatly improve the training efficiency on different computing devices with the premise of ensuring the registration accuracy.In summary,this dissertation focuses on the real-time and multi-modal slice-to-volume brain image registration methods based on deep learning.The key issues of slice-to-volume registration are systematically researched,which realize real-time and high accuracy registration without initial values and markers under the multi-modal scenarios.Furthermore,image dimensionality reduction and network compression are studied to improve the training efficiency for deep learning strategies.The research works in our study will lay a solid foundation for brain tumor detection and treatment.