Research On Visuo-Haptic Augmented Reality System For Hand Functional Rehabilitation

Visuo-Haptic augmented reality technology allows users to perform haptic interaction in a scene combining virtual and real.The application of visuo-haptic augmented reality technology in hand function rehabilitation training can not only render rich interactive scenes but also provide multi-sensory stimulation for patients,so as to improve their enthusiasm and initiative to participate in the training,and enhance the effect of rehabilitation training.The paper,based on the National Natural Science Foundation of China,develops a hand function rehabilitation training system using the natural feature-based visuohaptic augmented reality technology,and designs a rehabilitation training project of different difficulty levels.The research contents of this paper mainly include the following aspects:Firstly,the research status of visuo-haptic augmented reality technology is analyzed and summarized.Based on this,this paper proposes a natural feature-based visuo-haptic augmented reality system.This system uses ORB(Oriented Fast and Rotated BRIEF,ORB)and KLT(Kanade Lucas Tomasi,KLT)algorithms to match and track the target,and perform virtual registration of the haptic pen in the scene based on the rigid transformation of the visual haptic space.Secondly,to offer a more realistic interactive experience,this paper proposes a visual-tactile interaction method with 6-DOF(Six Degree of Freedom,6-DOF).According to the spatial conversion relationship,the method maps the posture of the haptic pen in the haptic space to the camera space,or projects the real-time motion postures of the haptic pen into the visual space,thus achieving a more realistic and natural human-computer interaction.The effectiveness and reliability of the 6-DOF visual-tactile interaction algorithm are verified through experiments.Finally,this paper develops and designs a hand function rehabilitation training system based on natural features of visual and tactile augmented reality.The system combines visual-tactile space conversion and 6-DOF visual-tactile interactive algorithm,and designs three rehabilitation training items with different difficulty levels: fixed trajectory,digital sorting and pickup training.The three rehabilitation training programs are mainly to improve patients' ability to move freely in their hands and upper limbs as well as their tactile perception.Finally,the feasibility,stability and effectiveness of the system are verified by experimental tests and questionnaire evaluation.