Knowledge And Data Co-driven Intelligent Recognition And Assessment Of Chinese Zither Hand Shapes And Fingerings

There is currently a surge of interdisciplinary development between science and arts with artificial intelligence being massively employed in music training and education.Chinese zither(Zheng)is a highly representative Chinese traditional string-plucking musical instrument.However,the traditional education model cannot provide real-time feedback on Zheng practice,which easily leads to non-standard hand shapes and fingerings,thereby seriously affecting the effectiveness of Zheng learning.Therefore,the intelligent and professional recognition and evaluation of Chinese zither performance techniques have become increasingly prominent in Zheng training.However,the requirements for Zheng performance techniques are difficult to be quantified as proper standards for intelligent classification and assessment,and datasets are scarce and difficult to collect.Namely,there are many gaps in the above disciplinary in terms of research paradigms,theoretical algorithms,dataset establishment,etc.In order to break through these professional knowledge and technical barriers,this paper integrates computer vision technology with Zheng fingerings,driven by a combination of image,video,audio data,and Zheng professional knowledge.Focusing on core issues such as scale formulation,dataset construction,and theoretical algorithm put forward,this paper explores the research paradigm of intelligent recognition and professional evaluation of Zheng hand shapes and basic fingerings.The main research and innovation are briefed as follows:On the one hand,the research paradigm of Zheng hand shape intelligent recognition is designed.Driven by image data and Zheng hand shape knowledge,the evaluating of Zheng hand shape is mapped to the image recognition problem of hand posture.(1)In order to solve the problem of difficulty in recognizing fine-grained hand images,a Zheng playing hand recognition method based on deep layer convolutional block attention module(DL-CBAM)is proposed.The specific work includes:integrating the key points of Zheng hand shapes,we formulate a Chinese zither hand shape classification scale,thereby providing a basis for subsequent intelligent hand shape recognition;In order to enable the network model to fully learn the features of hand shape images,a Chinese zither hand shape dataset is constructed using free view-angle acquisition and image enhancement techniques;Aiming at the characteristic that the intra class differences of fine-grained hand images are greater than the inter class differences,an attention mechanism is introduced to make the network model focus more closely on key information of hand posture.The experimental results show that this method can effectively recognize the Zheng hand shape.Compared to classical image recognition networks,it overcomes the problem of significantly increasing complexity after image data expansion,while improving the recognition rate of Zheng playing hand shapes.(2)In order to solve the problem of a sharp decrease in the classification and recognition rate of Zheng hand shapes caused by mirror hand images in actual self/other shooting modes,a Zheng hand shape recognition method based on multi-scale fusion double-layer network is proposed.The specific work includes: constructing a mirror hand shape dataset for Zheng based on the formulated Zheng hand shape classification scale;By designing a hierarchical network to achieve coarse and fine classification of hand images,and introducing multi-scale feature map fusion methods,the learning efficiency of the network is improved and the complexity of the model is reduced.The experimental results show that both the two-layer network and the multi-scale feature map fusion strategy can effectively improve the recognition rate of fine-grained hand images in different shooting modes.In addition,a new viewpoint generation technology and a naked eye stereoscopic display platform have been utilized to achieve three-dimensional display of Zheng hand shape,improving the receptivity of intelligent assisted recognition of Zheng hand shapes.On the other hand,a research paradigm for intelligent evaluation of Zheng fingerings is designed,driven by video,audio data and Zheng fingering knowledge,to transform the evaluation of basic Zheng fingering into a computer representable,computable,and interpretable skill assessment problem.(1)In order to solve the problem that the fine movements of guzheng fingering videos are not analyzed deeply enough from a professional perspective,a fingering assessment method based on Zheng technique logical discrimination is proposed.The specific work includes: developing a visual assessment scale for Zheng fingering based on artistic fingering requirements,which is used as a basis for subsequent Zheng fingering recognition;To remove the bottleneck of lack of fingering datasets,an interpretable dataset for basic right hand fingering of Chinese zither is constructed;We design an intelligent evaluation solution for Zheng fingerings,which includes three modules: hand recognition,key point tracking,and fingering assessment.At the same time,we deeply integrate prior knowledge of Zheng techniques to complete the auxiliary evaluation of Zheng basic fingerings.The experimental results show that this scheme can achieve the rationality evaluation and explanation of the internal movements of the Zheng basic fingerings.(2)In order to solve the problem of chord accuracy of traditional pentatonic instruments for chromatic scales,a Zheng fingering audio-visual comprehensive evaluation method based on multimodal decision level fusion of audio and video is proposed.The specific work includes: developing a Zheng fingering evaluation scale based on visual and auditory perception,and constructing a Zheng fingering audio and video dataset;To make vivd the characteristics of Zheng ‘right hand picking up sound and left hand picking up rhyme’,a Zheng fingering assessment solution based on audio and video heterogeneous data analysis is designed;Aiming at the different importance of audio and video data for left/right hand evaluation,a Zheng fingering based audio-visual intelligent comprehensive evaluation method is proposed,which achieves weighted fusion of three evaluation results: image based hand shape evaluation,video based fingering assessment,and audio based intonation comparison.The experimental results show that this scheme can simultaneously solve the problems of inability to determine pitch deviation solely based on visual evaluation,as well as the inability to trace the cause of fingering due to a single auditory evaluation,thus achieving complementary advantages between the two in fingering evaluation.This paper is innovative in exploring in an interdisciplinary field of Chinese zither,information processing,and artificial intelligence in an attempt of achieving intelligent recognition and evaluation of Zheng playing hand shapes and fingerings.This research approach has significance for academic and real world references as to the intelligent evaluation of piano and other instrument performance techniques.