Signature Verification And Writer Identification Based On Deep Learning And Domain Knowledge

Handwriting is an important behavioral biometrics.Personal identity verification and recognition based on handwriting has many application scenarios in today's society,and it generally includes the two following tasks: handwritten signature verification and writer identification.The former task has a broad prospect in administrative,banking and commercial applications,while the latter plays an important role in forensic document examination and historical document analysis.Handwritten signature verification is a challenging task,because signatures of a writer naturally exhibit a large intra-writer variability and may be skillfully imitated by an imposter.Furthermore,as signatures are difficult to acquire and signature databases are generally smallscale,research in this field relies heavily on feature engineering and has not fully explored the potentials of deep learning models for signature representation learning.Writer identification is still challenging when dealing with complex backgrounds and unconstrained handwriting styles,e.g.,historical document writer identification.Under this scenario,most of the popular techniques fail to capture the diverse handwriting styles effectively to achieve a high identification accuracy.To address the aforementioned issues,this thesis combines the use of deep learning models and domain knowledge and presents novel methods for signature verification and writer identification as follows:(1)To address the issue that dynamic time warping(DTW)relies heavily on feature engineering and selection in dynamic signature verification,we propose an end-to-end trainable deep soft-DTW(DSDTW)method,which enhances DTW with the capability of representation learning and greatly improves the verification accuracy.DSDTW uses convolutional recurrent neural networks to learn deep time functions as inputs for DTW;because DTW is not fully differentiable with regard to its inputs,we introduce its smoothed formulation,soft-DTW,to achieve better end-to-end optimization.Besides,as DSDTW performs local representation learning and does not need to model long-term dependencies,it can be effectively trained in low data regime.(2)We propose a deep learning-based dynamic signature verification framework,Syn Sig2 Vec,to address the skilled forgery attack without training with any skilled forgeries.Syn Sig2 Vec first applies the Sigma Lognormal model to synthesize signatures with different distortion levels for genuine template signatures,and then learns to rank these synthesized samples in a learnable representation space with supervision information from the signature synthesis process.The representation space is achieved by a novel 1D convolutional neural network,which is equipped with multi-head attention with learnable queries and extracts fixed-length representations from dynamic signatures of arbitrary lengths.(3)We propose a fractional max pooling network(FMPNet)for offline signature verification.FMPNet performs deep local representation learning and achieves superior results to two influential deep learning models,namely Sig Net-F and Sig Net-SPP.It maintains a large feature map to detect various local structures and employs spatial pyramid pooling to obtain global feature vectors,allowing a fine-grained description of signatures.Furthermore,we have collected a large-scale offline Chinese signature database with nearly 4,000 writers and 100,000 signatures.On this database,FMPNet achieves EERs of 6.84% and 0.10%against skilled forgeries and random forgeries,respectively.(4)To address the issue that historical documents generally have complex backgrounds,unconstrained handwriting and misleading writer annotations,we propose to encode Pathlet and SIFT features with bagged VLAD for historical writer identification.First,page-level rotation correction and U-Net-based deep binarization are suggested herein to obtain reliable binary document images.We propose a novel Pathlet feature to describe handwriting contours in a coarse-to-fine manner based on the path signature theory,and propose unidirectional SIFT to capture key local structures.To address the problem that a large codebook sparsely spreads out the data points and leads to a degraded performance,a novel encoding method,named bagged VLAD,is further proposed to effectively encode Pathlet and SIFT features.We achieve state-of-the-art results on ICDAR 2017 and 2019 historical writer identification database s without need of writer annotations for training.In the above studies,domain-specific knowledge,such as DTW,the Sigma Lognormal model and the use of handwriting contours,plays an important role in the proposed methods,indicating that a suitable combination of deep learning and domain knowledge can address existing issues in signature verification and writer identification more effectively.