Research On Storytelling-supporting Mixed Reality Environment And Its Interactive Technologies

Storytelling is a social and cultural activity where people share stories and has been proven to support the development of writing,reading,vocabulary,listening and other language skills.The storytelling in the mixed reality environment is a combination of digital information and physical characteristics,which directly displays the story,makes the story more interesting,and further enhance people's interest in storytelling.Nowadays,the technology of mixed reality is becoming more and more mature,and the traditional way of interaction cannot adapt to the interaction scene of virtual and real fusion.Tangible interaction enhances physical objects on the computer by coupling physical objects with digital data,making interactions more natural and convenient.Adding tangible interactive tools to storytelling will make the story livelier and more interesting,stimulate interest,and provide rich imagination.However,the current tangible interaction in Storytelling lacks similarity between virtual and real objects.This mismatch between virtual objects and real objects will reduce the seamless experience of mixed reality environments.In addition,current visual-based tangible interaction technology has high requirements for real objects in terms of feature points or data sets.Gesture is a natural and intuitive means of human-computer interaction.Using gestures to interact with virtual objects in a mixed reality environment can achieve naturalness.However,currently available mixed reality devices on the market,such as HoloLens,offer a single form of gesture,which requires the coordination of gaze points,and is not suitable for the direct control of three-dimensional objectsHandicraft production is a traditional cultural form that can improve children's practical ability and spatial imagination,and has become one of the main contents of the current primary and secondary school classrooms.The starting point of this paper is to use children's handicrafts as tangible interactive tools,to support children to create,design and tell stories in a more natural way by combining gesture interaction with natural desktop interaction,and to build a new storytelling model based on mixed reality technology.Taking the common origami as an example,we research and develop a mixed reality storytelling environment which combines origami production and tangible interaction.We conduct research on the problems of tangible interaction and gesture interaction,and provide natural interaction modes to support users' operations in the environment.In summary,the main work and contributions of this article are as follows1.This paper propose a new storytelling model based on mixed reality technology.Combining the two cultural activities of handicraft teaching and storytelling,it supports users to learn to make handicrafts,and makes use of handicrafts as tangible interactive tools,combining gesture interaction and touch interaction,to create and share stories on the desktop.Taking the common origami as an example,we develop a mixed reality storytelling environment that integrates origami making and tangible interaction,and provides a natural interaction mode for supporting users' operations in this environment.Results of user studies show that this mode can support users in handicraft production,story creation and storytelling,and stimulate users' interest in handicraft and storytelling Its natural interaction mode is easy for users to learn and use2.This paper proposes a physical interaction method based on real-time fusion of virtual and real objects.What users see through HoloLens is a virtual model over a real object.This method simplifies the problem of 6DOF pose estimation of objects in 3D space into the problem of object tracking on a horizontal desktop and the calculation of a rotation angle in 2D space.There is no need to establish a real object data set in advance,and there is no requirement for the number of object feature points.Experimental results show that this method can realize the matching of the virtual object and the real object in the position and the direction perpendicular to the desktop,and meet the real-time requirements3.This paper proposes a gesture interaction technology suitable for desktop mixed reality Storytelling.According to the requirements of supporting Storytelling to mix reality environment,two kinds of interactive gestures,namely desktop touch gestures and object grab gestures,are designed,which have a good sense of reality.For gesture data pre-processing,a gesture segmentation algorithm based on Kinect depth,color data and regional growth algorithm is proposed,which can segment gestures relatively completely A gesture classification model based on CNN is proposed for gesture recognition Experiments show that the proposed gesture segmentation algorithm can effectively improve the accuracy of gesture recognition,and the proposed gesture classification model has a high recognition accuracy and meets the real-time requirements.