Research On Stage Augmented Reality System Based On Multi-Kinect

Since augmented reality technology has the features of real-time interaction with the user and the high degree of integration of the virtual and real,it has been applied into various fields rapidly.However,such technology is rarely used into the stage performance field,the main reason is that the stage environment is complex,it is difficult for actors to interact with display system in real time.At the same time,the machine can't understand human behavior and unable to get information from the actors.Therefore,stage performance effects are usually obtained through video display,the actors need accurately match the video content in order to achieve the desired artistic effect,its seriously constrains the actor's performance.And the production of video effects needs high cost and long time.For the stage art innovation development,this is a profound significance and urgent task.To solve the problem,in this paper,we have researched and realize a real-time interactive system based on multi-Kinect.The system shows the corresponding rendering image according to the action of the actors in real-time,which completely liberates the shackles of the actors and coordinates the man-machine relationship of the actor and display system,making the stage effect determined by the actors rather than by the video passively.It is conducive to the development of stage performance art.Firstly,this paper deeply studies the key technologies of Kinect joint tracking algorithm,has a detailed understanding on the working principle of depth cameras,and does the depth test for the joint tracking of single-Kinect device,analyses and summarizes the existing problems.Secondly,aiming at the problems of limited identify scope,self-occlusion,joints' position vibration,unstably skeletal length of Kinect,the paper researches and realizes an approach for data fusion based on multi-Kinect.And the paper modifies the position and attitude transformation matrix between Kinects by using the ICP algorithm.Thirdly,aiming at Unity 3D engine characteristics and its development mode of component type,the paper designs and realizes a development framework based on message mechanism.Messages between the same layers are transmitted and processed directly,and messages from different layers are forwarded to the target layer by the message center.Fourthly,the paper has a deep study of the particle system and material system used by the rendering engine.With the combination of texture fusion and other means,the special effect system module is realized,the movement and change mode of particles are controlled by the original components and programs,and the effect is rich.At the last,combined with Unity 3D game engine,Photon Server API,OpenCV image algorithm library,Matlab DLL and Kinect SDK,we realize a stage augmented reality system based on multi-Kinect,which runs well.