Design And Implementation Of A Hand Motion Learning Assistant System With A Depth Camera

Standard fingering is a basic part of the initial learning of musical instruments.The lack of teacher's guidance at the beginning of the learning may result in fingering distortion or errors,which is difficult to correct after the beginners become accustomed,leading to reduced learning effects,increased learning difficulties,and even affect the beginner's enthusiasm for learning.In view of this real-life problem,this paper uses Guzheng fingering as an application scenario to design and implement a hand motion learning assistant system to aid in fingering practice for beginners of Guzheng.This system can demonstrate the basic fingering for beginners of Guzheng,and use the depth camera as input device to collect their playing fingerings,and compare them with the standard fingerings of the demonstration to indicate errors in the fingering,and so as to aid beginners on their self-exercises without teacher's guidance.Firstly,this paper introduces the relevant research background and significance.By consulting relevant research materials at home and abroad,the instrument learning assistance technology and hand recognition related technology are surveyed and summarized,their characteristics and deficiencies are summarized,and the research goals and objectives of the subject are determined research content.Then the paper introduces the related technologies which may get involved in the design and implementation of the system.In the third chapter,the user group of the system is defined based on the application scenarios assisted by the practice of Guzheng fingering,and the demand analysis of each function of the system is carried out.In the fourth chapter,this paper designs the overall architecture of the system,and divides into the image acquisition,data processing,pose estimation,fingering capture,fingering classification and evaluation,evaluation result visualization and data processing functional modules.A scheme based on hand pose estimation and dynamic time warping is designed.At the same time,the methods and attributes of the related classes of each module are elaborated in detail,and the interrelationship of each module of the system at work is specifically explained.In the fifth chapter,the function of each module of the system is tested and the test results are analyzed.At the end of this paper,the work of the project is summarized and the possible optimization points of the system are illustrated.The assistant system for fingering practice implemented in this paper can evaluate the users' fingering,provide reference comment for users' self-exercises,so as to solve the problem of limited teacher resources.In addition,the fingering evaluation scheme based on hand pose estimation and dynamic time warping proposed in this paper also provides a new idea for solving practical problems by using hand motion evaluation method.