Massive Video Processing And Face Recognition Application Based On Spark

With the advent of the era of big data and artificial intelligence,many fields such as national defense security,financial business,traffic management and other fields are developing towards intelligence,big data and cloud orientation.All fields will produce huge amounts of data,using the conventional data processing technology on the huge amounts of data processing causes low efficiency.Even the de veloper use the Hadoop core's distributed computing framework Map Reduce to deal with,efficiency analysis is still low.Therefore,new mass data processing and analysis solution technology are needed to solve the problem of massive data processing and low analysis efficiency.The main architecture of the new scheme includes Spark large-scale data computing,Hadoop HDFS,HBase distributed underlying storage,and message queue Kafka,etc.This article is combined with new solutions and technology to deal with massive video and the application of face recognition based on OpenCV,the essence is to use the Spark of mass data processing flow calculation of distributed parallel computing framework for massive video data processing,and based on Open CV image processing library custom Java process for face recognition.First,the video data is buffered to the Kafka buffer queue through the front end equipment,and according to the video key frame extraction method to convert video data into video frames.Then based on OpenCV image preprocessing the video frame image to pre-process video frames,enhance image attributes of the information needed in the process of face recognition.And then through the face detection of face image and non-face image,face posture optimization select good face image,eliminate useless images.Based on this,the feature extraction of SIFT face is carried out,then stored in the HBase and collected in the face database.Finally,the extracted feature vectors of face images are matched with the data of face database to realize the final process of the whole system.The new scheme can greatly improve the efficiency of mass data processing and analysis,and it has many advantages: architecture is more flexible,strong expansibility,convenient for late extension and iterative system upgrade,the latter e xpansion of business and management becomes more convenient.Based on the features of HDFS and Spark,the cluster can be built on a cheap and universal machine,which reduces the construction cost and maintenance cost of the cluster.Data resource stora ge has many forms,such as the diversified,structured,semi-structured and unstructured form,and it can manage the resource efficiency based on resource data mapping method.Based on Spark parallel computing technology,it can efficiently and quickly dea l with traditional hard CPU algorithm.