Swin-Spec Transformer Based Segmentation Research On Cholangiocarcinoma Microscopic Hyperspectral Images

Cholangiocarcinoma,as a malignant tumor with poor prognosis,has a low fiveyear survival rate.The diagnosis of cholangiocarcinoma relies on intraoperative or postoperative histopathological examination and requires a professional doctor to examine the pathological section under a microscope,which may be affected by the doctors’ experience,fatigue or subjective factors.With the rise of artificial intelligence technology,using artificial intelligence to process pathological images can make auxiliary diagnosis quickly,accurately and stably.At the same time,hyperspectral imaging technology also shows potential in the medical field.Hyperspectral images record rich spectral information at each spatial location,which can acquire spectral features that are difficult to observe with the naked eye and may further improve the potential of medical image processing.Semantic segmentation in cholangiocarcinoma microscopic hyperspectral images can help doctors quickly locate suspicious areas and reduce the workload of doctors.At present,Transformer-based vision networks have made great breakthroughs in color image segmentation tasks,but existing color image segmentation networks cannot make full usage of the rich spectral information of hyperspectral images.To this end,this paper proposes a Swin-Spec Transformer segmentation network,which treats hyperspectral images as a sequence of two-dimensional grayscale images and preserves spectral structure information.Based on the efficient Window-based Multi-Head SelfAttention,two types of windows are used to calculate attention and extract features in spatial and spectral dimensions,respectively.To obtain accurate two-dimensional segmentation results,this paper also proposes a spectral aggregation method based on spectral tokens,which summarizes the spectral features of each spatial location.Since cholangiocarcinoma is a rare disease,the number of doctors who have experience in this professional field is relatively small,and their time is limited.Thus,only few cholangiocarcinoma microscopic hyperspectral images have corresponding high-quality labels.However,directly using the low-quality labels will reduce the performance of segmentation networks,so how to fully utilize low-quality labels to further enhance the segmentation performance remains to be resolved.For this purpose,Label-to-Photo translation is introduced and this paper proposes a two-stage hyperspectral segmentation task deep learning framework based on Labels-to-Photo translation and Swin-Spec Transformer（L2P-SST）.In the first stage,the OASIS generative network and the Swin-Spec Transformer discriminative network are used for adversarial training,and a spectral perceptual loss function is proposed to generate high-quality hyperspectral images;in the second stage,the generative network is fixed and the generated hyperspectral images are used as data augmentation in the training of Swin-Spec Transformer segmentation network to further improve its performance.Using the proposed Swin-Spec Transformer segmentation network and the L2 PSST two-stage hyperspectral segmentation task deep learning framework,this paper achieves 76.16% m Io U（mean Intersection over Union）,85.80% m Dice（mean Dice）,90.96% Accuracy and 71.65% Kappa coefficient in the semantic segmentation task of cholangiocarcinoma microscopic hyperspectral images collected in our laboratory.Experiments show that our methods can effectively distinguish cancerous tissue from normal tissue,help doctors to achieve rapid diagnosis of cholangiocarcinoma pathological tissue,and accelerate the promotion of computer-aided diagnosis technology based on microscopic hyperspectral imaging for clinical practice.