| As a tool for abstracting,defining,and expressing problems,formulas are an indispensable part of scientific and technical documents.Offline handwritten formula recognition can convert handwritten formula images into LaTeX strings that can be edited and expressed by computers.It has a wide range of applications in the scenarios of smart education and daily life such as intelligent examination question search,intelligent marking,assisted input,and transcription of information,which can directly facilitate our learning,work,and daily life.However,traditional methods cannot correctly recognize handwritten formula images due to the complex two-dimensional symbol structures in formulas.With the development of deep learning,the researchers try to adapt the attention-based encoder-decoder model to recognize handwritten formula images and have achieved significant progress.However,with the increasing modeling capability of models,the overfitting problem has gradually become a bottleneck in this task.Besides,current work has not optimized for the recognition of long complex formula images,and the recognition performance of these formulas images remains relatively poor.As a result,the overfitting problem,as well as the challenges brought by long complex formula images,are urgent problems to be solved in this task.Focusing on the overfitting problem and the challenges that are brought by the long complex formula images in offline handwritten formula recognition,the research of this thesis is carried out in three aspects:training data,training strategy,and model structure.In order to improve the recognition performance of handwritten formula images,the data augmentation methods,the training strategy optimization methods,and the methods to enhance the ability of modeling symbol dependencies in formulas are studied in this thesis.The main contributions of this thesis are listed as follows:(1)The data augmentation methods for handwritten formulas images are studied.To address the problem of overfitting caused by the limited training data that can hardly support the training of complex recognition models,three data enhancement methods are proposed in this thesis:1)A symbol replacement-based handwritten formula image generation method is studied in this thesis,which can both expand the visual and semantic information of the source training samples through image transformation algorithms and the symbol replacement method.2)The construction of a handwritten formula image dataset that consists of the formula images in real life,this thesis has constructed and disclosed the largest handwritten formula image dataset HFID.Compared with the existing dataset in this field,HFID has the advantages of a larger data scale,more symbol classes,broader coverage,and closer to real application scenarios.3)A method of generating handwritten formula images based on generative adversarial networks is studied in this thesis,which is based on the idea of image style translation and can convert the printed formula images to handwritten formulas images in the target handwritten style.P2H-CycleGAN which can solve the problem of steganography in CycleGAN and P2H-GANwriting which fuses the multi-scale style and content features are proposed in this thesis to generate handwritten images with high quality.The experimental results show that data augmentation methods proposed in this thesis can improve the generalization ability of the model and alleviate overfitting.(2)The optimization method of handwritten formula image recognition based on training strategy is studied.Aiming at the problem that the existing training strategies do not optimize the model overfitting,which easily makes the model overfit in the training set,this thesis proposes a shuffled multi-round training strategy based on the idea of simulated annealing algorithm and Dropout.This strategy divides the entire training process into multiple training rounds.In each training round,the network parameters obtained from the previous training round are inherited.Besides,we randomly shuffle the symbol dictionary at the beginning of each training round to bring disturbance to the model.This training strategy enables the model not only to inherit the features learned in previous rounds but also to reduce the dependence on some of them and learn new ones,so as to enhance the generalization of the extracted features and alleviate overfitting.The experimental results show the effectiveness of the shuffled multi-round training strategy in reducing overfitting.(3)The optimization method for the recognition of long complex formula images through multimodal dependencies among symbols is studied.To address the problem that the implicit dependence modeling in existing models is difficult to learn long dependencies between symbols,this thesis proposes a multimodal dependence attention model.This module first merges the visual and semantic features of the symbols to enhance the ability of symbol representation for the model.Then,it uses the multimodal symbol features and an attention mechanism to learn the dependencies between the current symbol to be recognized and the symbols recognized in earlier time steps effectively.Finally,it uses the dependencies to assist the recognition for the current symbol,so as to achieve the purpose of improving the recognition performances of the model.The experimental results show that the multimodal dependence attention indeed learns the visual and semantic dependencies among symbols and improves the recognition results of long complex formula images. |
