Research Of Visual Interaction For Virtual Reality Flight Training

Flight simulator is an aviation equipment that can simulate airplane’s movement in air andground realistically. It is a typical application of system simulation technology and virtual realitytechnology which is widely used in civil aviation and military aviation areas. Since the beginning of21th century, the Flight Simulation and Advanced Training Engineering Technology Research Centerof Nanjing University of Aeronautics and Astronautics has been concentrated on the study of flightsimulator based on virtual reality technology and proposed the concept of semi-virtual reality cockpitin which the force and tectile sense are provided by real operation equipments and the visual sense isgenerated by HMD (Helmet-Mounted Display). The system also needs an appropriate way to providethe visual feedback of user’s hand, but it will affect immersion sense and destroy the naturalness andconcordance in interaction if traditional CyberGlove is used. Therefore, this paper adopts aninteractive scheme based on hand pose estimation to visualize the virtual hands. In addition, computervision technology is also applied to desktop virtual reality training system as a complementaryprocedure of training pilots. The main research contents are summarized as follows:1. Gesture features extraction and fusion. The gestures are constrained by the equipments insemi-virtual reality cockpit. In other words, the category of gesture and pose of hand are determinedby the shapes and positions of the equipments. An appearance-based method of hand pose estimationis utilized in this paper, which converts hand tracking problem into a hand pose indexing problem byconstructing gesture image database and creating a mapping between gesture features and hand poses.Before gesture features extraction, image preprocessing which contains hand segmentation, contoursearching, contour smoothing and skin filling is performed. Then the HOG (Histogram of OrientedGradient) features and chromaticity histogram features are fused using evaluation of retrievalprecision to identify the fusion weights.2. Large-capacity and high-dimension feature indexing. In order to improve retrieval speed andsolve the “Curse of Dimensionality” problem, an improved LSH (Locality Sensitive Hashing) methodis proposed in this paper. And a predictive model is built to evaluate the index parameters. Thesimulation results show that the predictive model is appropriate to the practical index performanceand the time consumption could be reduced by41.9%at the cost of the recall rate drop3%for10-NN(Nearest Neighbors) retrieval.3. Hand pose estimation under self-occlusion. According to hand self-occlusion insemi-virtual reality cockpit, multi-view gesture databases are introduced. In order to avoid capacity in databases increase excessively, a tree model is constructed using multi-view gesture features. UnderMAP (Maximum a Posteriori) framework, tree node searching is used to ensure the pose estimation isoptimal in probabilistic sense. Besides, the estimation results are weighted by temporal consistencywhich could enhance the accuracy. Experimental results show that the speed of the proposed methodcould be increased from4.4fps to8.2fps, and the poses of finger and wrist could also be estimatedaccurately.4. Fingertip detection, position and gesture recognition. According to the Desktop Virtual RealitySystem, contact and non-contact training methods are accomplished in this paper. The contact trainingmethod uses fingertip detection and position algorithms to judge the collision between real hand andvirtual objects. Fingertip detection is realized by combining mark identification and convexity defectsanalysis, and dual orthogonal camera system with image depth information is proposed to implementthe fingertip position. Experimental results show that the average accuracy of position could be asgood as0.1cm. In non-contact training, a gesture recognition method based HTM (HierarchicalTemporal Memory) network which is inspired by vision system of mammal is proposed. Recognitionrate of the proposed method is higher than90%against10kinds of gesture. Combining the gesturerecognition and fingertip position, a pilot could complete the whole process of flight simulation onlyutilize movement gestures.