Application Of Multimodal Perception Of Visual And Haptic Augmented Reality In Hand Rehabilitation

Augmented reality is a new technology that integrates real-world information with virtualworld information.In recent years,related research has integrated haptic perception technology into augmented reality technology,which can realize not only the fusion of real and virtual objects,but also visual and haptic perception.With the development of the new generation of information technology,visuo-haptic augmented reality technology has gradually shown its great influence in the field of hand function rehabilitation,and its advantages such as interestingness,interactivity,cost-effectiveness,and portability are increasingly favored by patients with hand dysfunction.In the rehabilitation process,patients can not only see and touch virtual objects in the real world,but also patients' diseased limbs can move with virtual objects and receive haptic feedback from virtual objects.This technology relieves the boring feeling caused by patients' multi-sensory treatment,and fully stimulates patients' haptic perception to achieve the purpose of upper limbs rehabilitation training and improve the effect of rehabilitation training.This research is funded by the National Natural Science Foundation of China,and develops a hand function rehabilitation system using visual haptic augmented reality technology based on Marker-SLAM.The main contents of this paper are as follows.In this paper,a haptic reconstruction method for the perception of the contour and surface texture of three-dimensional objects in a virtual environment is proposed.A hybrid collision detection and response algorithm based on domain monitoring is adopted to realize the haptic interaction of virtual objects.To display the attributes of the object surface,a texture force model based on image data is put forward.The height map is constructed using the improved Pentland shading restoration algorithm,and then the constraint force models in tangential and normal directions are established.Afterwards,the improved "bump mapping" technology is utilized to map the texture image onto the surface of the 3D virtual object.Finally,the texture force is applied to the operator through a haptic device to realize the perception of the texture and contour of the 3D virtual object surface.Experimental results show that the algorithm can effectively generate haptic texture information from images and improve the accuracy and realtime performance of haptic perception.This paper also proposes a visuo-haptic augmented reality interaction algorithm based on Marker-SLAM.Based on the Marker-SLAM tracking registration algorithm,an augmented reality environment is built,effectively solving problems such as immobile camera,poor stability of the virtual object registration,and the unmatched model dimension in virtual and in real-life.Furthermore,to add haptic sensor technology to the augmented reality environment,a non-tracker-based stylus tip posture optimization algorithm is proposed,and finally the visual haptic augmented reality interaction algorithm based on Marker-SLAM is completed.This method accurately calibrates the haptic spatial coordinate system and the world coordinate system,and realizes the installation and replacement of the haptic spatial coordinate system to the world coordinate system,the installation and replacement of the world coordinate system to the camera coordinate system,and the conversion of the camera coordinate system in turn to image coordinate system,and finally completes the fusion of the visual haptic space.Experimental results show that this method has high accuracy and stability.In addition,a visuo-haptic augmented reality hand rehabilitation system based on MarkerSLAM is designed and developed.The system mainly includes two stages of rehabilitation training,that is,sensory desensitization training and sensory retraining.Desensitization training uses a variety of contact methods to continuously stimulate sensitive areas,improve excitation of damaged nerve structures,and encourage the formation of new circuits.Whereas sensory retraining effectively trains the ability of the hands to hold objects,as well as wrist joint translation and rotation.Finally,the feasibility,stability,and effectiveness of the system are verified by analyzing the experimental data through statistical methods and questionnaires.