| Augmented reality(AR)is a technology that enriches and improves people's visual experience by seamlessly integrating computer-generated virtual information into real scenes observed and seen by users using video display equipment.It is a new and important way of human-computer interaction.In recent years,it has been widely used in science and technology museums,multimedia classrooms and other teaching occasions,but most of them interact through AR technology with handheld signs,which has strong limitations and poor usability and immersion.Therefore,it will be the development direction of human-computer interaction in the future to take people as the input of AR instead of handheld signs.In particular,taking human hands as the input of interaction can effectively improve the defects of handheld signs.Based on the augmented reality teaching task in the field of electronic components instrumentation,by analyzing and improving relevant technologies at home and abroad,an AR system based on the interaction between human fingertip and virtual components was developed and applied to the teaching process of electronic components instrumentation.The key is the tracking and fingertip detection of the hand part,the 3d registration of the unmarked fingertip,and the human-computer interaction between the fingertip and the virtual components in the plug-in environment by judging the position relationship between the fingertip and the midpoint of the system identification.Its specific contents include:(1)Aiming at the problems of limited registration environment and unfriendly interaction in the traditional 3d registration method based on the marker board or fixed markers in the environment,this paper proposes a method to realize 3d registration by using the natural features of unmarked fingertips,which is more flexible and more natural than the traditional registration method.With the combination of CamShift method and skin color detection algorithm,the whole hand area was completely captured to complete the real-time tracking and detection of the hand part.In the aspect of fingertip detection,an adaptive fingertip detection algorithm for extracting K value was proposed to realize the accurate detection of fingertip position,and the validity of the algorithm was verified by experiments.(2)Study the 3d registration of unlabeled fingertip based on the homography matrix.In the registration of the initial frame,the feature points were first detected between the hand plane and the hand template plane in the initial frame by theimproved FAST algorithm,and the registration was completed by using the nearest neighbor algorithm combined with RANSAC for matching.In the subsequent stage of3 d registration,the homography matrix is obtained by matching features between each frame and the previous frame.At the same time,a feature point re-detection strategy is proposed to improve the robustness of 3d registration algorithm and avoid the error of projection matrix to accumulate with the increase of the number of frames.Finally,the hand model was constructed to obtain the world coordinates of the fingertips and complete the 3d registration of the fingertips.(3)Developed the virtual component instrumentation teaching application system based on the natural characteristics of unlabeled fingertip,which was based on augmented reality technology and applied the fingertip based interactive method.By adopting modular design scheme,it has certain portability and applicability.The system allows users to use their fingers to perform real-time virtual component mounting and interactive operations,which develops the way of HCI in the AR system and improves the immersion of HCI. |
