The Research And Implementation Of Realtime Virtual-reality Interaction Technology In Mixed Reality

In recent years,with the development of computer graphics technology and related hardware devices,the mixed reality(MR)field represented by Holo Lens and Magic Leap devices has ushered in a vigorous development.Mixed reality has gradually entered the vision of the majority of users and has been widely used in entertainment,education,military,medical and other fields.Unlike virtual reality which constructs a fully virtualized three-dimensional world and augmented reality which superimposes virtual objects onto the real world,mixed reality has received more and more attention due to its unique characteristics of retaining reality in virtual and transforming reality into virtual.Mixed reality technology is a comprehensive technology that integrates human-computer interaction,computer vision,computer graphics,artificial intelligence and other disciplines.In the field of mixed reality interaction research,the main research interests are gesture recognition,remote assistance,multi-user interaction,virtual-reality interaction,etc.Most of the existing virtual-reality interaction technologies are based on manual markers or preset three-dimensional models to superimpose dynamic virtual objects in a static real scene,or to interact with virtual objects by recognizing hand information.A few studies do not rely on specific preset objects,and directly introduce real object information as an interactive tool like gesture information,voice information,and eye movement information into the process of virtual-reality interaction.Therefore,in this thesis,computer vision algorithms are used to identify and track real objects without relying on preset marks and CAD models,bringing real objects into the process of virtual-reality interaction as an interactive tool.Specifically,the main research work of this thesis is as follows:(1)Aiming at the problem of insufficient user participation in the existing mixed reality interaction technology,a method of using real objects as interactive tools is proposed,and a virtual-reality interaction framework with the participation of real objects is constructed.This allows real objects to affect virtual objects,and use real objects to interact with virtual objects more naturally.The virtual-reality interaction framework constructed in this thesis is based on Holo Lens.After theoretical analysis and experimental verification,a complete solution for virtual-reality interaction development is proposed.It mainly includes four modules: user intent analysis module,mixed reality capture module,Holo Lens communication module,mixed reality interactive module,and put forward an effective implementation plan for each main module.(2)Aiming at the problem of image markers and 3D models in mixed reality technology that need to be preset in advance,two real-time virtual-reality interaction methods that do not rely on markers and prefabricated models are proposed.One is real-time tracking interaction based on target detection and target tracking,the other is real-time modeling interaction based on instance segmentation and reconstruction.Through computer vision algorithms,both methods achieve that real objects interact with virtual objects without any preset objects.(3)Aiming at scenes that only need the three-dimensional position information of real objects to participate in the interaction,a real-time tracking interaction method based on target detection and target tracking is proposed.This thesis uses the target detection algorithm YOLO v3,the target tracking algorithm CMT and the spatial mapping grid to realize the three-dimensional positioning of the real object,thereby realizing the real-time interaction between the real object and the virtual object based on the three-dimensional position.In addition,for the problem that the CMT algorithm lacks too many feature points of the object,it will cause the tracking loss.A multi-thread-based auxiliary correction method is proposed in this thesis,which can still keep track of the real object when a single CMT algorithm fails to track.(4)Aiming at scenes that require specific contour information of real objects to participate in the interaction,a real-time modeling interaction based on instance segmentation and reconstruction is proposed.This thesis uses the instance segmentation algorithm YOLACT to segment the image frame to separate the target object,and then reconstructs the virtual2.5D replicas of the target object in Unity,and calculate the position and pose according to the depth information collected by the Holo Lens RGBD camera,so that the 2.5D replicas is rendered in the corresponding positions in the mixed reality scene,thereby realizing the realtime contour-based interaction between real and virtual objects.