Embedded GPU-accelerated Panoramic Video Stitching And Object Detection Strategy

Panoramic video is a sort of video shot at the same point of view to record the full scene.The collecting devices of panoramic video are getting widespread attention with the development of VR and live-broadcasting video technology.Nevertheless,CPU and GPU are required with the strong processing abilities to make panoramic video.The traditional panoramic products depend on large equipment or post processing,which results in high power consumption,low stability,unsatisfying performance in real time and harmful to the information security.In addition,applications of intelligent cameras are needed in the security field and the sports broadcast,in which the target detection technology is the basis of the intelligent camera.Therefore,the target detection of panoramic video can expand the detection range and provides the basis for the later intelligent analysis.The main work of this thesis is as follows:(1)We propose a L-ORB(Local ORB)feature detection algorithm.The algorithm optimizes the feature detection regions by segmenting the video images and simplifies the scale invariance and rotation invariance.Then the features points are matched by the multi-probe LSH algorithm and the progressive sample consensus algorithm is used to eliminate the false matches.Finally,we get the mapping relation of image mosaic and use the multi-band fusion algorithm to eliminate the gap between the video.The L-ORB algorithm we proposed can shorten the feature extraction time to 1/3 of the traditional ORB algorithm and 1/1000 of the SIFT algorithm.(2)We optimize the YOLO(You Only Look Once)target detection algorithm in the calculation scale and network structure,by analyzing the existing convolutional neural network.And we utilize the 8-bit integer instead of 32-bit floating point for deducing the neural network.Moreover,the fusion network structure to further accelerate the speed of the network is derived.Through the optimization of the YOLO algorithm,the speed of target detection has increased by more than 3 times.(3)In addition,we utilize the NVIDIA Jetson TX2 embedded system that integrates ARM A57 CPU and Pascal GPU,with its Teraflops floating point computing power and built-in video capture,storage,and wireless transmission modules,implementing the multi-camera video information Real-time panoramic splicing system.And we effective use of GPU instructions block,thread,flow parallel strategy to speed up the image stitching algorithm.The experimental results show that the algorithm has good performance in the feature extraction and feature matching of the image splicing.The algorithm speed is 11 times that of the ORB algorithm and 639 times that of the SIFT algorithm.The performance of the system is 29 times higher than that of the traditional embedded system,but its power consumption is as low as 10 W.(4)We implement the robot autonomous navigation system by using ROS system and Cartographer algorithm package in the open source robot platform TurtleBot.By using two NVIDIA TX-2 embedded chips,we have tested our proposed panoramic stitching and target detection strategy.The results show that the algorithm proposed in this thesis can meet the requirements of real-time splicing and target detection.