Research And Implementation Of Distributed Processing Platform For Parallelized Surveillance Video Data

With the construction of smart city,video surveillance is widely used in all walks of life.The scale of surveillance video system is growing,the resolution of surveillance cameras is becoming clearer,and the amount of video data has grown dramatically.In order to mine more valuable information from the large-scale surveillance video data and assist the surveillance personnel to realize intelligent video surveillance,more and more video analysis algorithms are introduced and used,which lays more weight on the computing load of the surveillance video system.Faced with massive video data and various complex video analysis,how to flexibly couple video analysis algorithms to an efficient video data processing platform has become an important challenge.On the other hand,in the current era of data explosion,distributed technology has become a technology hot spot in recent years.The emergence of distributed technologies such as Hadoop and Spark has provided effective solutions for solving massive data processing and storage problems.In view of the above problems in the field of surveillance video,this dessertation proposes a platform based on the distributed parallel processing of surveillance video data.The main research contents include:In the platform architecture design,the platform is mainly composed of three parts: message center module,analysis and calculation module and storage module.In order to solve the problem of high coupling between video processing algorithm and system physical resources in surveillance video system,this dessertation uses Kafka message queue as the message center module to collect video data and cache intermediate result data.As the middleware,different video analysis modules in the platform are decoupled to improve the flexibility of the platform.The video analysis module is decoupled to increase the flexibility of the platform.In order to meet the demand for near real-time processing of large-scale video data,this dessertation uses Spark distributed computing engine as the analysis and calculation module to be responsible for the efficient and parallel video analysis and calculation of video data.In terms of parallel processing of video data,in order to better realize the parallel calculation of video analysis algorithms,based on the idea of distributed data parallel processing,this dessertation divides video analysis algorithm into two categories: inter frame correlation and inter frame independence.It also proposes a data division and reading strategy based on repeated frames.For these two types of algorithms,Kafka partition storage and Spark pull data are designed with a customized data access strategy.The implementation of video data processing is based on the highly parallel distributed platform proposed in this dessertation.Finally,based on the platform designed in this dessertation,performance testing and evaluation are conducted through case applications.The test results show that the platform proposed in this paper has the ability to process large-scale data,the calculation speed is fast,and it can flexibly couple various video analysis algorithms on the distributed cluster,with good scalability and flexibility,which brings convenience to the later platform expansion and management.