Research On Parallel Processing Strategy And Application Of Video Big Data Based On Spark Framework

As an important data source,video data has the typical characteristics of huge volume,fast speed,sparse value and completely unstructured data,which is one of the important research contents in the field of large data technology.In order to get real-time video data processing and analysis results,improving the processing performance of large video data has become one of the key issues to be solved in large data technology.Spark is a memory-based parallel computing framework that greatly improves the performance of large data processing through memory computing.This paper takes the elevator video big data as the research object,and puts forward a parallel processing strategy of video big data based on Spark framework,which provides key technical support for smart elevator based on video analysis.The main contents of this paper include:(1)According to the characteristics of large video data and elevator environment,a parallel processing framework for large video data based on Spark framework is designed,which includes video data distribution layer,video data transmission layer,video data processing layer and video data application layer.The framework can be used for elevator video monitoring in distributed environment,and real-time processing and analysis of collected elevator video.(2)According to the characteristics of typical video analysis algorithms that have no inter frame dependency,we proposed a parallel strategy based on spark framework.Video streaming data is divided into independent data blocks by data set parallel mechanism,which improves the efficiency of algorithm analysis by expanding nodes.Taking the elevator passenger number analysis algorithm as an application case,the validity of the parallelization strategy is proved.when the number of nodes is 2,5and10,the performance is improved by 176%,248%,471%.(3)According to the characteristics of typical video analysis algorithms that have inter frame dependency,we proposed a parallel strategy of video inter frame correlation analysis algorithm based on spark framework.By using the pipelining parallel mechanism,the dependency relationship of video analysis algorithm is established,and the efficiency of algorithm analysis is improved by combining steps and threads.Taking elevator door anomaly detection as a typical case,the validity of the parallelization strategy is proved.when the number of nodes is 2,5and 10,the performance is improved by 143%,161%,238%.Taking elevator video monitoring as the application object,we developed a elevator intelligent monitoring prototype system based on Spark,including elevator passenger number monitoring module,elevator door anomaly monitoring module,elevator drop object monitoring module and elevator distribution visualization module,which effectively improves the processing performance of multi-channel video streams,and lays a foundation for future intelligent analysis of elevator video data.