Mixing Of Virtual And Actual Reality Method For Augmented Reality Aided Products Assembly Guidance

Augmented Reality(AR)is a new type of human-computer interaction technology.It can superimpose virtual world information into the real world through computer simulation,which is perceived by human senses and enhances the real world.Since AR technology has the ability to simultaneously display real world information and virtual world information,it has been increasingly applied to the assembly process of complex products such as aerospace and space in recent years.The assembly guidance technology based on AR technology can greatly improve the operator's understanding and efficiency of the assembly process of mechanical products by superimposing the computer-generated assembly guidance information(including 3D models,assembly animations,text prompts,etc.)in the assembly site.Mixing of virtual and actual reality is the core to realize the ?seamless? integration of virtual guiding information and assembly site.However,most of the current AR-guiding system for product assembly simply superimposes the virtual cue information on the actual assembly environment image captured by the camera,and does not deal with the virtual and real fusion relationship between the real object and the virtual information.This not only causes the user to have visual fatigue,affects the experience,but also causes the assembler to be ambiguous about the understanding of the assembly position,reduce assembly efficiency,and increase the probability of error.Although some scholars have proposed some solutions to the virtual and real fusion problem in the assembly process,there are still some problems such as the applicability and robustness of the virtual and real registration method for the mechanical assembly environment,and the virtual and real fusion in the assembly interaction process is difficult to deal with effectively.In view of the above problems,this paper analyzes the characteristics of the mechanical product assembly environment in detail,and conducts in-depth systematic research on the registration and virtual and real occlusion of virtual guidance information in the assembly environment during the virtual and real fusion process.It aims to eliminate spatial location ambiguity of virtual guiding information and enhance the realism of virtual guidance information in the mechanical product AR assembly environment and thus improve the practicality of the system.The primary research work of this paper is as follows:(1)Edge-Snapping based depth image restoration methodHigh-quality depth image is the basis of virtual and real registration work.Considering the influence of object surface characteristics and material properties on depth image,we combine the color channel image information to reduce the depth of image noise,during which process noise kernel function and dependency kernel function are introduced to restrain the noise diffusion to a non-noise region.Secondly,the color image is also used for snapping the object boundary in the depth image because of similar object boundaries between them,and the improved Fast Marching Method(FMM)is used to fill the cavity region,so that good boundary characteristics of the object in depth image can be achieved.Comparing with the existing methods of depth image denoising,the proposed method has better depth image quality and higher object boundary precision,which lays a good foundation for point cloud generation.(2)Point cloud and visual features fusion-based virtual-real registration method.Registration and tracking technique is prerequisite for the mixing effect in augmented reality.To improve the applicability and robustness of the tracking method for augmented reality-based mechanical products assembly,a point cloud and visual features-based tracking method is proposed.Aiming at the characteristics of mechanical products assembly environment,a 3D model point cloud based tracking coordinate system definition method is presented.Then point cloud combining with visual features is used to improve the robustness of tracking process.To improve the number of visual feature matching points in texture-less assembly scene,a visual feature matching strategy based on the consistency of direction vectors is proposed.Finally,the loop detection and global pose optimization are added to the tracking process,which improves the applicability and robustness of the tracking method for the AR assembly of mechanicalproducts.(3)LINE-MOD template matching based rigid object occlusion handling method.Occlusion handling is a crucial index to achieve mixing effect in augmented reality.The emphasis of this chapter is the occlusion handling of rigid objects.Firstly,the reference images from different view angles are collected in CAD environment in the offline phase,and the corresponding camera posture of each reference image is recorded.In the online phase,the reference image which is most similar to the current view image is obtained by template matching,and its corresponding pose is obtained.Then,the accurate posture of the camera is precisely estimated by interpolation calculation.Finally,the corresponding CAD model is superimposed on the surface of the object to be assembled according to the tracking results,and the virtual and actual occlusion of the object to be assembled and the guiding information is realized by judging the spatial position between the models.The experimental results show that the method has strong usability,good real-time performance,and high accuracy.(4)Active Appearance Medel based non-rigid object occlusion handling methodGiven the existence of the non-rigid object(hand)in the mechanical assembly process,an approach for dealing with the occlusion relationship of non-rigid object and guiding information based on depth calculation is designed by using the active appearance model contour tracking algorithm.The correct virtual real occlusion relation is obtained by comparing the depth value in the segmented region and the depth value of the virtual assembly guiding information.Comparing with the existing virtual and actual occlusion methods based on depth calculation,the proposed method avoids the problem of large computational complexity caused by the comparison of depth values on the whole depth image and improves the real-time and practicability of the system.Based on the above researches,a prototype system of occlusion handling method for augmented reality aided products assembly system(CPILab-AAPS(Cyber-Physical Interaction Lab-Assembly Assistance Prototype System)is developed.Meanwhile,some experiments are conducted to validate the feasibility and effectiveness of the proposed methodologies and approaches.