Multi-Modality Medical Imaging Synthesis Task For Positron Emission Tomography Imaging

Positron emission tomography(PET)is a functional imaging method that can visualise the metabolic level and biological activity of diseased tissues in the body,and it is important for the early detection and clinical diagnosis of many diseases.Compared with computed tomography(CT)and magnetic resonance imaging(MRI),PET images have the disadvantages of low spatial resolution and poor image quality.Therefore,PET imaging always is combined with CT or MRI imaging technology,forming the popular PET/CT or PET/MRI imaging systems.However,the hybrid PET/CT or PET/MRI imaging system also suffers from some new problems,such as extra CT doses of ionizing radiation and the long data acquisition time due to MRI.Motivated by the development of deep learning techniques,many CNN-based methods are proposed to generate CT or MRI from PET data for avoiding extra CT or MRI scanning for patients.In 2019,Xue Dong et al.successfully converted the non-attenuation corrected PET(NAC PET)images to the corresponding CT images using a cyclic consistent generative adversarial network(CycleGAN).In 2020,Changhui Jiang et al.have successfully generated high-quality pseudo-MRI images via generative adversarial networks(GAN).Despite these benifits on the "one-to-one" modality transformation,few studies explored the multi-modality medical imaging synthesis task based on PET imaging,such as the PET-CT&MRI imging synthesis.In practice,both CT and MRI scans are required because of the complementary information between MRI and CT modality.In this case,we present a convolutional neural network based on an alternating learning strategy to synthesize CT and MRI from PET input data.In this work,we replace the multi-decoder architecture with modality signal,switch layer and alternating learning strategy for multi-modality imging synthesis.The proposed network makes full use of the parameter redundancy between multiple learning tasks,requiring only a small number of parameters to distinguish between these tasks.In this work,the modality signal for PET-PET dataset is set to 001 code,that for PET-CT dataset is set to 010 code,and that for PET-MRI dataset is set to 100 code.And then,the modality signal is fed to switch layers,and is encoded to adjust feature maps in the decoder in order to generate the target modality(determined by the modality signal).We can maually set the modality signal to predict the desired modality images.The experimental results demonstrate that the proposed network is available to generate MRI and CT images with high quality,even if in the lack of reference images.Taking PET-CT data as an example,the experimental results show that the predicted CT images have high contrast,clear structure,nealy zero noise and have a very high similarity to true CT images.Quantitatively,the mean absolute error(MAE)is smaller than 25 HU value between the predicted CT and true CT images,the peak signal-to-noise ratio(PSNR)is higher than 30 dB,and the structural similarity metric(SSIM)is closed to 90%.Moreover,the predicted MRI images from PET-CT dataset have also high quality.Although the predicted MRI images cannot be evaluated quantitatively due to the lack of the MRI labels,the radiological experts give high scores to the pMRI images in noise rejection,detail recovery and overall quality.This work can generate multimodal images containing clear anatomical structure information,which benifits to disease diagnoses.Besides,the work using the modality signal,switch layers and alternating learning strategy dramatically reduces the number of parameters and keep the strong capacity for generating images,which has positive implications for other multi-task learning in the image field and has important practical applications.