Multi-modality Liver Image Synthesizing Using Liver Vessel Segmentation Label

The acquisition of medical image samples is crucial in clinical analysis and diagnosis,expert system construction,and teaching and training of novices.At the same time,medical image images samples can be used as training samples for deep learning models,which are also significant for downstream intelligent analysis in supporting diagnostic tasks.Nowadays,medical image acquisition is a labor-intensive task that consumes a lot of human and material resources,especially for 3D medical images.In addition,the process of medical image acquisition is often subject to privacy disclosure,medical ethics,and data copyright issues.Recently,deep learning(DL)-based generation models have been successful for image synthesis tasks.Such models can synthesize new pseudo-images that match the real style based on the feature distribution of the training image dataset,which are random and diverse while having texture and structural features that correspond to the real image.Because of this property,image generation models provide a new solution for the expansion of medical image data.DL-based image generation models can synthesize specific medical images based on random vectors or specific input conditions that match the current requirements.In this paper,we propose a multi-task generative adversarial network and a multimodal liver image synthesis for 3D liver images based on 3D liver segmentation image data,combined with the principles of existing medical image number generation models,as follows:(1)Two 3D liver image synthesis datasets were constructed for the first time.The first dataset is a 3D liver vessel segmentation mask and a liver contour mask.The second dataset is a multimodal dataset of 3D CT and 3D MR images.(2)3D liver vascular information has important anatomical structural features of liver organs.In this paper,we combine the structural features of liver organ textures and blood vessels to explore the possibility of using 3D liver vascular segmentation labels as input to synthesize 3D liver images.(3)To address the issues such as the poor stability of existing generator models and the lack of guidance from a priori medical knowledge.We propose a multi-task(i.e.,segmentation task and generation task)3D generative adversarial network for 3D liver CT image synthesis.We utilize the semantic mask of the liver as a priori medical knowledge to guide the generation of 3D CT images,which can reduce the computation of a large amount of background and thus make the model more focused on the generation of the liver region.In addition,we introduce a stable multi-gradient descent algorithm optimization method in the model to balance the weights of the multitasking framework.To the best of our knowledge,this is the first application for the 3D liver CT image synthesis task.Experiments were conducted on a real collected dataset and the experimental results showed that the Dice similarity coefficient for the segmentation task reached 0.87 and the performance of the multitask synthesis model outperformed the existing state-of-the-art methods.(4)We propose to extend the designed multitask generation method into a multimodal synthesis model to extract the common features of MRI and CT modalities to improve the accuracy and robustness of the model.Extensive experiments were conducted on the constructed multimodal dataset,and the experimental results show that the proposed multimodal approach outperforms existing methods in both quantitative evaluation and qualitative analysis.