Research On Key Technologies For Augmented Semi-virtual Reality Cockpit

Compared with traditional flight simulators, VR based flight simulator has great potentialities inthe applications for its compact structure and relatively low price. As the level of technology andhardware restrictions, there were also low performances of human-computer interaction in this type ofsimulator. It is difficult or unable to complete the some complex interaction in cockpit. In order tosolve this problem, we developed an augmented semi virtual reality cockpit technology to achieve anatural human-cockpit interaction. The main contents of this article can be summarized as follows:1.Augmented semi-virtual reality cockpit. There are several shortcomings in VR based flightsimulator: no real visual effect of3D model hand, electromagnetic tracking device is susceptible tothe interference of the environment, no real haptic and etc. An augmented semi-virtual reality cockpitsystem is proposed, using image based rending hand to replace the geometric model, using opticaltracking system to detect head pose, design of semi-physical cockpit to ensure user’s haptic feedback,to build a an augmented semi-virtual reality cockpit environment with real visual feedback, tactilefeedback, haptic feedback. The experiments showed that our method can decrease by50%cost timethan the traditional method without data gloves and electromagnetic tracker. It can significantlyreduce hardware and software development costs and enhance system availability.2.Adaptive hand segmentation and visualization of virtual hand. In order to solve the susceptibleto interference problems of skin color detection for hand segmentation, the continuous pixels changesbetween the video frames is used to updated parameters of the elliptical color model in real time, toachieve optimal parameter estimation. The experiments show that this adaptive hand segmentationalgorithm was more robustness than the traditional algorithm. In order to make more accurate andsmoothing hand contour, the bilateral filtering algorithm is taken to smooth2D curve, which cansmooth the contour and also maintain the sharp features. Because the poor visualization of3D handmodel, hand image was fusion to realize real visualization of virtual hand. Experiments showed thatvirtual hand’s movement and texture were same as the user’s real hand. The user would completeinteractive task more quickly in the immersion environment.3.Head tracking and registration based the orthogonal camera video positioning system. To solvethe problem that electromagnetic tracking device is susceptible to the interference of the environmentand with high cost, an orthogonal camera video positioning system that the optical axes of camerasare set along axes of an orthogonal coordinate is proposed, which all cameras are pointed to the originof coordinate. Then we realize a quick and accurate iterative localization algorithm to compute theobject’s position. Though the theoretical derivation and analysis of conversion relationship betweenthe multi-coordinate system in the registration process, we can select optimum measurement data. As the error is reduced as much as possible, accurate registration is achieved in the augmented virtualsystem, to meet the requirement of interaction.4.Effective VR modeling based CAD data. The geometry modeling is a foundation work invirtual reality system. In traditional manual modeling way, large amount of modeling work and themodel dimensional accuracy is difficult to guarantee. The CAD model was exported to the media fileand then the mesh data was simplified through the sorting algorithm. Base on quadric error metrics(QEM) algorithm, introducing quadric compactness as new vertex’s generation to simplify complexgrid. Experiments showed that triangular mesh shape by our algorithm is more reasonable than QEMalgorithm in the same simplified conditions, with higher quality of rendering. The mesh data could begenerated with a level of detail VR model by setting different simplified threshold. When the meshwas simplified90%, model shape had no change, and the rendering frame rate could be raise onetimes above.Finally, an augmented virtual reality flight simulator prototype system is built to test theperformance of human-computer interaction in our system. The subjective experiment showed that thevisualization of virtual hand is obviously superior to the traditional3D model hand, its appearancefeatures (flexible deformation and skin texture) is actual same as real hand. The objective experimentshowed that users can complete interaction task more quickly than other interactive method.