Research On Multi-modal Mixed Reality System Based On Holo Lens And Haptic Rendering

With the development of computer technology,human-computer interaction(HCI)technology has become an indispensable part of daily life.As an important part of HCI technology,haptic rendering provides operators with a way to actively explore and recognize the virtual world.In the traditional haptic rendering devices and methods,there is a certain difference between HCI and the real world interaction such as friction and inertia between mechanical structures that unavoidable.In order to enhance the qualities of traditional haptic rendering technology and achieve the spatial consistency of haptic rendering with visual rendering,this paper aimed at a non-contact,visual and haptic fusion mixed reality system based on the Microsoft HoloLens intelligent holographic glasses and haptic rendering technology.First of all,this paper theoretically investigates the generation and control principle of electromagnetic field.Then the technological feasibility of haptic rendering based on electromagnetic method is analyzed.In additional,the formation mechanism of mixed reality technology with respect to Microsoft HoloLens intelligent holographic glasses is illustrated.Secondly,the overall hardware scheme of multi-modal mixed reality system based on HoloLens and haptic rendering is designed.The hardware of the system mainly includes three parts: the visual rendering module based on the HoloLens,the haptic rendering module based on the electromagnet and permanent magnet,the human hand position detection module based on the Kinect.The whole hardware construction process of the system is introduced from the aspect of hardware selection to determination of structure parameters.Finally,on the basis of hardware system,the function of each module is realized by software programing.These functions include virtual environment rendering based on HoloLens,finger position detection based on Kinect,electromagnet excitation signal control based on TM4c1294 board and data communication among modules.In order to obtain the relationship between the position of permanent magnet and the electromagnetic force it receives,ANSYS finite element simulation method is used in this paper to analyze the magnetic induction intensity and the electromagnetic fore of the permanent in the operating space when a single electromagnet is applied with the excitation current.According to the result of the simulation analysis,verification experiments are carried out to prove the correctness of the simulation.Based on the relationship between the position of permanent magnet in the operating space and the electromagnetic force,the optimal control method of electromagnetic force was proposed,and the duty and direction of excitation current applied in six electromagnets were calculated by offline simulation data to generate controllable electromagnetic force.In order to verify the effectiveness of the multi-mode mixed reality system designed in this paper,the virtual object recognition experiment based on the electromagnet module,multimodal reappearance authenticity assessment experiments and tactile perception experiment are carried out.It can be seen from the experimental results that the multi-modal mixed reality system designed in this paper can realize the multi-modal human-computer interaction,and achieve the spatial consistency of haptic rendering and visual rendering.Based on the HoloLens mixed reality technology and electromagnetic tactile rendering technology,this paper designs a non-contact multi-modal mixed reality system with visual and tactile fusion,which lays an important foundation for natural-oriented and non-contact HCI technology.