Research And Implementation Of Large-scale Real-time Face Retrieval System And Task Scheduling Algorithm

With the development of the Internet and the popularization of intelligent electronic products,the amount of available data has increased significantly,which makes the processing technology of big data emerge and develop rapidly.At present,the methods of processing big data are divided into two categories,one is real-time streaming processing,and the other is batch processing.Image-based face recognition is a typical application of big data processing,and the research on high recognition accuracy algorithm has been developed systematically.However,the real-time face recognition and retrieval technology under large-scale face image data has just started,which needs to be studied and solved urgently.This paper studies the technical difficulties on real-time face recognition and retrieval under large-scale face image data,designs and implements an autonomous,controllable,efficient and concurrent system platform,which provides application examples and reference significance for real-time processing of large-scale image data.The main work of this paper is as follows:1)This paper analyzes the requirements of the large-scale real-time face retrieval system on running workload,data storage capacity and real-time retrieval performance,and finally confirms the distributed processing scheme.After analysis and testing,the real-time streaming processing platform Storm is selected.2)This paper designs distributed dynamic row-key storage scheme.Analysis of the characteristics of single camera 10 millions capturing image data per year and single table 10 millions unstructured person information is carried out.In order to improve the retrieval efficiency and ensure the integrity,the frameworks of large-scale data distributed storage are compared and analyzed.A targeted distributed dynamic row-key storage scheme is designed based on HBase.3)This paper designs the profession-platform loosely coupled computing interface.In order to embed the big data professional computing and face recognition algorithms into the distributed platform more flexibly,the profession-platform loosely coupled computing interface is designed and the JNI interface specification is defined.For the purposes of replaceable,professional and dynamic,the distributed concurrent dynamic scheduling mechanism is designed.4)This paper designs and implements a large-scale real-time face retrieval system.The face recognition algorithm is embedded into the distributed platform,and the complete process functions of face capture,photo catch,face detection,feature extraction and face retrieval are designed and implemented.The speedup ratio of the system is tested with 50,000 photo data in a single process,and the maximum speedup ratio can reach 21.05.Finally,the system is running under the actual data size of 500,000 photos and the average face retrieval time is 1.77 seconds,which verified the feasibility and real-time performance of the system.