Research On The Classification Of Tongue Features Of Traditional Chinese Medicine Based On Deep Learning

In recent years,social problems such as an ageing population,sub-health and social stress have become increasingly serious,and there has been a significant increase in people’s commitment to their own health management.Tongue diagnosis is one of the most important diagnostic tools in Chinese medicine.It is an intuitive and convenient way for Chinese medicine practitioners to understand a patient’s health condition and make a diagnosis by observing the characteristics of the tongue.However,this diagnostic method relies on the theoretical knowledge and clinical experience of the Chinese medicine practitioner,which is subjective and unstable.Therefore,it is an inevitable trend for the development of tongue diagnosis to promote digitalisation and objectivity based on modern technology.With the rapid advances in artificial intelligence technology,combining deep learning technology with traditional Chinese medicine tongue diagnosis theory,researching two key issues of tongue segmentation and quantitative analysis of tongue features,and establishing an intelligent diagnostic model for Chinese medicine tongue diagnosis is a feasible means to realise digital and objective tongue diagnosis.The main work of this thesis includes the design of a pre-extraction method for the tongue region,the design of a tongue segmentation method based on traditional image processing techniques and deep learning techniques,the design of a multi-feature classification model for tongue images based on deep learning,and the study of visual interpretation of model decisions.The thesis has been developed in accordance with the following four areas of research:（1）Pre-extraction of tongue region based on Dlib facial feature point recognitionBased on the feature that the original tongue image captured by the tongue acquisition device contains a complete human face,this paper uses the Dlib face 68 feature point detector to obtain the corresponding pixel point coordinates of the lip and chin feature points around the tongue.The tongue region is then pre-extracted according to the interval ratio of these coordinates.This method improves the accuracy of tongue segmentation and avoids the wastage of arithmetic power caused by redundant background pixels and errors in the production of tongue segmentation datasets.（2）Tongue segmentation model designBased on traditional image processing techniques,the tongue foreground marker and background marker are implemented using the Grab Cut algorithm,and the tongue segmentation is achieved using the masking method.In addition,the tongue edges were labeled,and a tongue segmentation dataset with a total of 482 samples was made according to the VOC format,and the training and test sets were divided 9:1.A tongue segmentation model based on the UNET semantic segmentation network was constructed,and VGG16 was used as the backbone feature extraction network to improve the network structure and enhance the training effect of the model based on migration learning.In the end,the Pixel Accuracy（PA）and Mean Intersection over Union（MIo U）of the tongue segmentation model based on UNET reached 98.54% and 97.14% respectively,with high segmentation accuracy and robustness.（3）Design of a multi-tongue feature classification model based on Res NetThe classification datasets of 11 tongue features such as tongue color,fissures and tooth marks were produced based on the theory of Chinese medicine tongue diagnosis,all of them with a total of 482 samples,and the same 9:1 division of the training and testing sets.The Res Net-34 network was used as the backbone feature extraction network to build the classification model for each tongue feature,and the training efficiency of the model was accelerated based on the migration learning method.Finally,the classification recognition of11 tongue features was achieved.The models have good classification effect for 10 tongue features other than tongue color,with Accuracy（Acc）and F1-Score reaching more than 83%and 74%.However,the classification effect of the tongue color feature model is poor and still needs to be improved afterwards.（4）Decision visualization of a multi-tongue feature classification model based on Grad CAMBased on the Grad CAM method,the last convolutional layer of the backbone feature extraction network in the multi-tongue feature classification model is processed to obtain the importance weights of the neurons in the convolutional layer based on the back-propagation gradient,and a map of feature-sensitive regions is obtained by linear weighted summation to visualise and interpret the decisions made by the multi-tongue feature classification model.