Handwritten Mathematical Expression Recognition Based On Structural Features

Handwritten mathematical expression recognition is to convert human handwritten mathematical expression into a format that can be understood and edited by the computer.It is widely used in the fields of paper document electronization,content recognition and retrieval.With the deepening of social informatization and intellectualization,handwritten mathematical expression recognition for intelligent education has become one of the research hotspots.Handwritten mathematical expression recognition includes offline recognition and online recognition.The offline handwritten mathematical expression recognition first obtains the image containing the expression by scanning and then recognizes the expression from the image.Handwritten mathematical expression has complex spatial structure and semantic dependence between symbols.Handwritten mathematical expression recognition not only needs to detect and recognize each symbol in the expression but also analyze the structural relationship between different symbols.Therefore,high-performance handwritten mathematical expression recognition is a very challenging task.In this thesis,the structural features and semantic dependency of handwritten mathematical expression are deeply studied.On this basis,a handwritten mathematical expression recognition algorithm based on structural features is proposed and improved.Firstly,the deep learning networks is used to detect and locate the symbols in the mathematical expression.Secondly,the structural relationship between symbols is modeled by the graph neural network.Finally,the mathematical expression recognition is realized by a decoder based on Transformer.The main work of this thesis is as follows:1.Locate the symbol position in the expression.In order to construct the relationship between symbols,it is necessary to accurately locate the position of symbols.This thesis studies the symbol detection performance of different detection networks through comparative experiments and selects the network with better comprehensive performance as the detection module of this algorithm.2.A handwritten mathematical expression recognition algorithm based on graph neural network is proposed.In this thesis,the expression is regarded as a graph structure with two-dimensional spatial layout.Each symbol in the expression is abstracted as a node of the graph and the relationship between symbols is abstracted as an edge of the graph.Based on the location information of symbols,the graph representation of expression is constructed according to the LOS(Line-of-Sight)rule.In order to make full use of the structural information of the expression,a graph reasoning network based on node-edge attention is proposed to learn and update the two-dimensional structural information of the expression,and recognize the expression by classifying nodes and edges.3.An improved handwritten mathematical expression recognition algorithm based on graph neural network and Transformer is proposed.Handwritten mathematical expression is essentially a language text with contextual semantic information.Therefore,it is necessary to integrate the structural information and semantic information of the expression to improve the performance of the model.Therefore,a decoder based on Transformer is designed to decode the features encoded by the graph reasoning network and learn the semantic information among the symbols,so as to generate the recognition results.In addition,in order to make full use of the forward and backward semantic information of the expression,a bidirectional semantic modeling strategy is adopted to further improve the performance of the algorithm.The recognition rates of the proposed algorithm are 53.45%,55.27%,54.13% and66.75% respectively on the public datasets CROHME 2014/2016/2019 and Off Ra SHME,which verify the effectiveness of the proposed algorithm.