Design And Implementation For Virtual Reality System That Combined With Multi-Channel High-definition Video And Gesture Recognition

In the process of vigorous multimedia technology development,multimedia application system based on virtual reality technology occupies an important position.As one of the popular VR applications,the VR system combined with multi-channel high-definition camera and gesture recognition device creates a stable,super nice and immersive experience for users.Also this is very import for the realization of UAV remote rescue,remote survey,remote video conferencing and other applications.This paper mainly focuses on the design and development of this VR system and deeply digs into the working principle and implementation of multi-channel high-definition video stream reception and pixel format conversion,gesture recognition and mixed pipeline programming with 3D graphics rendering.Firstly this paper researches the efficient method of acquiring multi-channel high-definition video picture and video pixel format conversion,which lays the theoretical foundation for multi-channel video stream creating RGB texturel.Then,it researches the mapping implementation for gesture recognition and action instruction,which provides the realization method for the multi-channel high-definition camera control.Next,it researches the mixed pipeline programming with 3D graphics rendering,which provides the basis for the rendering of virtual scene.In addition,it completes the design of virtual reality system and realizes the system initialization,multi-channel high-definition video stream reception and transcoding,user input and virtual scene rendering.Finally it runs a functional verification test and performance test for this virtual reality system with comprehensive test framework which defines extensive samples for various system submodules.In the process of designing and implementing this system,the paper mainly focuses on system computational resources load balance problem during video pixel format transforming and virtual scene rendering.Aiming at this issue,it studies the video pixel formate transform method which bases on FFmpeg and shader programming.Using FFmpeg,the implementation is simple.But during the whole transforming,it only optimize in a certain degree,the computing pressure of the whole system is still in CPU.Using shader programming,the transforming can be carried out in the GPU parallel computing unit,which realizes the load balancing for the computing resources.Therefore,this paper designs and uses a 3D graphics hybrid rendering pipeline to achieve highly efficient capabilities for video pixel format transforming and virtual scene rendering.The experimental results show that this VR system has high efficiency in rendering and stable performance.Also this system has been applied to the Datang Mobile 5G performance demonstration project,and attends many well-know technology exhibitions and with Datang Mobile.