The interaction methods of non-wearable is a relatively natural interaction method in the flight cockpit,covering multiple emerging fields such as mixed reality,computer vision,and human-computer interaction.The paper chooses the "human-oriented" natural human-computer interaction method.The paper abandons the application of data glove monitoring and the computer vision mixed reality scheme that based on mark points,and adopts the virtual hand mixed reality interaction scheme that based on the natural feature points of the hand.The solution uses a real flight simulator to complete the virtual presentation of accurate hand postures,giving the operator a "real touch,virtual vision" experience to satisfy the sense of reality and immersion during flight training.The content of the paper is as follows:1)The paper calibrated the flight cockpit mixed reality system that based on the see-through helmet display.Combined with camera calibration technology,the internal and external parameters of the camera are calculated,and the corresponding relationship between the 3D coordinates of the object in the cockpit and the 2D coordinates in the image is obtained to complete the system calibration in the flight cockpit.2)A hand detection and segmentation algorithm based on skin color and key feature points of the hand is proposed.First,using YCbCr space based on Otsu adaptive threshold algorithm to detect skin color,adapt to the influence of different image brightness.Secondly,the hand key point model is used to quickly detect the skin-like regions.The algorithm to eliminate the interference of other skin-color regions.3)The paper constructs a three-dimensional tracking and registration system.The system based on hand natural feature points in the flight cockpit.First,The system uses the hand color histogram feature to track the hand.Second.The system combines the artificial fish school algorithm to optimize the prediction of the particle filter algorithm.finally,The paper realize the registration scheme that based on the match between the current frame of the hand and the key frame.4)A gesture recognition method based on key points of the hand and skin color is proposed.First,the paper extracts the contour features of the Fourier descriptors connecting the skeleton of the hand feature points.Second,the recognition combine the optimization capabilities of the artificial fish school algorithm,and to optimize the support vector machine model parameters to improve the recognition rate.5)The virtual hand presentation in the flight cockpit is completed by Hololens2 glasses equipment.After the hand is tracked and recognized,the real coordinate position of the hand and the gesture model are obtained,and then a fusion 2D gesture map is added to the virtual cockpit,which is presented on the personnel view through HoloLens.The mixed reality hand interaction system in the flight cockpit solves the problem that hand detection is susceptible to the complex background interference in the cockpit environment and the interference of different illumination;it solves the problems of particle shortage and failure,and greatly improves the tracking effect;combined with camera calibration technology Complete the three-dimensional registration of the virtual hand in the world coordinate system in the cockpit;solve the problem that the Fourier descriptor feature is easily affected by the background,the gesture of the hand,and the single gesture expression,which greatly improves the accuracy of recognition and classification.Finally,during flight simulation,hand interaction can meet the needs of pilots and bring the operator a realistic and immersive flight simulation experience. |