Design And Implementation Of Surveillance Video Stream Data Collection System Based On Kafka

In recent years, with the rapid development of Internet, big amount of data generated by human activities is growing explosively due to the decline in the price of storage devices and the increasing number of electronic devices.A significant increase in data leads to the rapid development of big data technology. On the other hand, the rapid development of video surveillance technology, as well as its wide application in various industries, makes the surveillance video stream become typical big data. The surveillance video stream has unique characteristics compared with text stream. For example,the generating velocity is stable, and the size of data stream is large, and the demand of orderly is high. These features make it an important challenge to capture surveillance video streams.Aiming at this challenge, this paper designs and implements a video stream collection system based on Kafka. We take Kafka's advantages of high throughput and low latency. In order to improve the scale of collection and achieve scalability, we design and implement a distributed architecture for the collection system. In the aspect of resource scheduling strategy, we propose a method which allows the cluster to dispatch tasks according to the distribution of Kafka partitions. This solution reduces the internal network load between nodes and improves the performance of system.Experiments indicate that the system is capable of collecting large-scale surveillance video stream, and it has high efficiency and practicability.