Fish Position And Orientation Detection Method For Mixed Reality Fish Tanks

Using the projector to project the virtual scene to the back wall of the real fish tank,a Mixed Reality(MR)fish tank can be constructed that integrates the virtual ocean world and the real fish tank environment.Research has shown that this system can help attract the attention of children with Autistic Spectrum Disorder(ASD),who can concentrate better in this system,interact with the ocean world and swimming virtual sea creatures,and learn cognitive lessons.Sea creatures can be made to imitate and follow the movement of real fish to enhance the sense of immersion and interaction,so as to design more interesting course content and achieve a better effect of virtual and real fusion.This requires real-time and accurate tracking and calculation of 3d swimming trajectory and posture of real fish.But most of the existing methods for tracking fish are not suitable for fish tanks.Reflection and refraction in fish tanks increase the difficulty of tracking real fish compared to underwater environments such as oceans and lakes.Some works only keep video sequences of fish swimming in the central volume of the tank,ignoring the reflections of fish swimming in the side walls of the tank.Others pasted white paper around the tank to avoid reflection.In addition,as far as we know,there is no research on the calculation of fish direction in 3D space,although there is a small amount of work described using a camera to calculate the direction of motion in a 2D top view.Therefore,this paper proposes a real-time method to track the position and posture of a single fish in a fish tank:two cameras are placed on the top and side of the fish tank respectively to obtain the video image of the fish;In the case of reflection,the position of the fish in the two views is calculated,and the rotation angle of the fish in the two views is calculated by ellipse fitting method.The information from the two views was then combined to calculate the position and orientation of the fish in real time,taking into account the refraction of the water.Experimental results show that this method has good accuracy,robustness and real-time performance.The method presented in this paper has been applied and validated in the MR fish tank based learning system for children with ASD.To sum up,the main work and contributions of this thesis are summarized as follows:1.A method is proposed to calculate the 3D position of fish in real time by combining information from bi-orthogonal views,which captures real-time fish movements by placing cameras on the top and sides of the tank.First,we combined the background subtraction and tracking algorithm to obtain the two-dimensional outline of the fish in the two views,and then judged the reflection situation and dealt with it according to the situation.If the fish body and the reflection are separated,the real contour is directly screened according to the positional relationship.If the fish body and the reflection are stuck together,the offset correction of the contour position is required to obtain the centroid position of the real contour.Finally,considering the influence of water refraction,the three-dimensional position of the fish is calculated in real time using ray tracing.The experimental results show that the method proposed in this thesis has higher speed,accuracy and success rate than the method proposed by Muller et al.Taking 1000 frames of images as test samples,the average frame rate of the method proposed in this thesis is 61.78fps,the accuracy is 85.2%(the accuracy is set to 5 pixels)and the success rate is 84.3%(the coincidence rate is set to 0.8).2.A method is proposed to calculate the 3D orientation of fish in real time by combining the information of bi-orthogonal views.We use an ellipse to fit the body of the fish according to the morphological characteristics of the fish.In the top view,we judge the direction of the fish head according to the position of the center of mass of the fish body,take the angle between the short axis of the ellipse and the horizontal line as the rotation angle of the fish body in the top view,and use the correlation between the two orthogonal views to calculate the rotation angle of the fish body in the side view finally uses the spatial linear mapping relationship to calculate the three-dimensional orientation of the fish in real time.The experimental results show that,with 1000 frames of images as test samples,the accuracy of the method proposed in this thesis is 70.2%(the difference between the calculated value and the marked value is within 0-10°).3.An early childhood education system based on MR fish tank is given,which uses the three-dimensional position and three-dimensional orientation of real fish to drive virtual submarine creatures to follow the movement of real fish in real time,so that virtual marine creatures in the virtual world can interact with fish moving in the real world.Realize interesting course content and build a more vivid mixed reality system.First,we smooth the obtained 3D data,and then establish data communication to transmit the data to the Unity engine.Then unify the Unity world coordinate system and the real world coordinate system,and finally assign the processed three-dimensional position and three-dimensional orientation data of the real fish to the virtual seabed creatures in the virtual scene,so that they can swim with the real fish.