Detection Of Mathematical Formulas In Print Document Based On Deep Learning

It is a practical and useful work to use the computers to comprehend mathematical formula from images of documents.However,mathematical formula in documents are often surrounded by natural language texts,extracting and comprehending the formula in documents are a difficult task.It is thus a useful and practical work to develop a system of mathematical formula recognition for a variety of natural language texts,and also a challenge to the intellectual.Extraction of formula from the images of documents is the first step to develop such as system and there remain many problems in this area of research.The research of extracting mathematical formula from texts falls mainly into two categories: 1.the extraction of mathematical formula from isolated areas;2.the extraction of formula mixed with texts.The research mainly focuses on documents written in Chinese and English.Different methods are used to extract formula under different backgrounds.Extraction of embedded formula and isolated formula also needs different methods.So far,there has not been a single method suitable for both embedded formula and isolated formula.Some problems remain for mathematical formula extraction,and these include:1.the erroneous extraction of the item numbers and equation numbers as formula;2.the erroneous recognition of English words as formula in Chinese texts;3.the failure of extracting the formula in tables.A method for formula extraction is proposed in this thesis.This method possesses excellent capability of generalization.This method treats formula as a special category,and employs the same approach for both Chinese and English.This is also applicable to other languages.The Faster R-CNN method of deep learning is chosen for formula extraction after comparing target detecting neural nets such as YOLO,SSD,etc.Image libraries are constructed by labeling formula in a large set of document images in both Chinese and English,and these images are used for training the Faster R-CNN deep nets.After many rounds of modifications of the neural net topology and training parameters,as well as an improved data library,a deep net with very high recognition rate is realized.A method for automatically generating library images is proposed in order to reduce the effort of manually labeling the formula in the images.This method is effective in the generation of library images.The Faster R-CNN is modified to overcome the inaccuracy problem in the position of extracted formula.Through the integration of multiple Feature Maps,these Feature Maps are compressed into a unified space for hyper features.Candidate areas are formed on hyper Feature Maps,and the positioning accuracy is improved.In the last step,the formula in the picture is automatically intercepted by using the coordinate file obtained by the target detection.A set of 2379 formula contained in 200 randomly selected document images were used to test the modified Faster R-CNN neural net and formula recognition rate of 96.51%was achieved and formula extraction accuracy of 91.76% was achieved.Experimental results show that our method effectively solved some of the problems in the literacy.