Research Of Stave-Aware Optical Music Recognition Technology For Camera-Based Scenarios

With the rapid development and wide application of smart phones,music intelligent information services based on singing evaluation and sight-singing practice are more and more used by people.For existing singing applications,songs and musical scores are usually built-in,and it is difficult for users to freely take pictures and import musical scores.However,the existing music score recognition software is not ideal for the recognition of music scores imported from photos.Therefore,this paper intends to carry out research on the recognition of printed music scores in the real camera-based scenarios.This paper proposes a complete musical score recognition method based on staff line perception by adopting the musical notation object detection method.This method can directly input a complete picture of music score,after the staff detection and musical notation detection and classification processing,and finally through the semantic reorganization of musical notation,the position,pitch and duration of notes are output line by line.The main contributions of this paper includes:(1)Due to the lack of printed music score dataset for camera-based scenarios,the Camera Printed Music Staves(CPMS)dataset has been self-built and published and the spectrum curvature,angular distortion and uneven light that may occur in real camera-based scenarios are fully considered;(2)A staves line perception model based on object detection is proposed for the first time,which introduces the detection of staves on the basis of object detection of notes,which effectively improves the recognition accuracy of note pitches,and has good adaptability for the existing music score bending problem in camera-based scenarios;(3)Implement the recognition system for photographed printed music scores,including staff image preprocessing module,musical notation and classification module and musical notation semantic reorganization module.Finally,our method achieves 99.23% pitch accuracy,97.17% duration accuracy and 96.59% note accuracy on the CPMS test set,all of which exceed the detection accuracy of the state-of-the-art sequence recognition-based methods.At the same time,compared with the sequence recognition method,our method does not need to take pictures of music scores to participate in training,and has stronger generalization.Aiming at the problem of unsatisfactory detection accuracy of printed music scores due to the bending of staff lines,omplex combination of musical notation semantic rules and uneven light in camera-based scenarios,this paper proposes a music score recognition model based on staff line location,which effectively integrates the information of the location of the music score,the position and classification of the music notes,and obtained the best recognition accuracy of note pitch and note duration at present.