| With the rapid development of information technology and deep learning technology,the application scenarios of handwritten mathematical expressions have increased greatly,which requires fast,stable and accurate handwritten mathematical expression recognition methods.However,because of the complex two-dimensional structure of the expression itself,the irregularity of handwritten characters,and the lack of stroke information in offline scenarios,handwritten mathematical expression recognition has become a challenging task.In recent years,global recognition methods based on deep learning have emerged in the problem of handwritten mathematical expression recognition,which is represented by encoding and decoding structures.This type of method does not require separate character cutting,recognition and structure analysis.However,there are still problems in this method at present,such as the lack of feature extraction ability of the existing traditional convolutional network,the difficulty of training caused by the timeseries dependency during the training of the recurrent neural network,and the problem of poor structural generalization.Research in this field still needs to be explored continuously.In summary,the main contents of this article are as follows:(1)For the handwritten mathematical expressions input by pictures,a serialized expression recognition method Res-MSA based on the encoding-decoding structure is proposed.First,the input image feature sequence is extracted using a residual network-based convolutional neural network,and then the multi-head attention in the Transformer network is used to analyze the correspondence between the feature sequences,and predict character by character to generate the current result.This method is trained in parallel,and the convergence speed is faster than that of the traditional encodingdecoding model using a recurrent network.The correct expression recognition rate of 48.6% is achieved on the CROHME2019 competition data set,which proves the effectiveness of this method.(2)Aiming at the difference in character size and distribution position of the expression part in the handwritten mathematical expression image,a serial multi-scale image feature extraction module is proposed.Through serial fusion of feature maps of different resolutions in the Res Net network,the purpose of including semantic information and structural information in the image is achieved.This method effectively solves the difficult problem of identifying complex expressions in the Res-MSA model,and achieves a correct expression recognition rate of 50.25% on the CROHME2019 competition dataset.(3)Aiming at the structural characteristics of mathematical expressions,the mathematical characters are classified according to their substructures,and a tree-like expression method is proposed.Based on tree structure representation and multi-head attention,this paper proposes a tree decoding model.The tree decoding model firstly predicts the parent node,and the parent node helps predict the child node,and finally analyzes the branch relationship between the parent and child nodes to complete the prediction of the current node.The tree decoding method achieved higher recognition performance than the serialized decoding model,and achieved a correct recognition rate of 53.43%of expressions on the CROHME2019 competition dataset. |
