Research On Smart Glove And Its Key Algorithms For Middle School Chemistry Experiment

During the COVID-19 pandemic,numerous students were required to take remote courses from their homes.This approach presents countless challenges,especially concerning offline chemistry and biology experiments that are inherently hazardous and therefore difficult to conduct remotely.Moreover,several secondary school chemistry experiments require extensive use of materials and involve dangerous experimental processes,such as strong acids,explosions,flames,and so on.As a result,many experiments cannot be effectively operated remotely and even many observable experimental phenomena,such as the production of bubbles and generation of water mist,are challenging to observe during the experimentation process.Virtual experiment platforms offer an effective solution to both issues.Nevertheless,using either the WEB approach with a mouse and keyboard or VR and AR reduces students’ authentic experimentation experience and hampers their hands-on ability.What’s more,unfamiliarity with these new experiments makes it challenging to remember the experimental rules and conditions,making it difficult to successfully complete them.With the advancement of XR technology,many virtual experiment platforms have gradually adopted mixed reality-based virtual-reality fusion interaction where physical devices interact with virtual objects,ensuring immersion.However,existing MR experiment platforms present the disadvantages of having redundant electronic hardware devices and hazardous security.These interaction methods are also excessively reliant on indoor interactive input devices,resulting in the obscuring and failure to recognize the user’s tiny movement gesture changes and tiny experimental objects.Therefore,this research proposes MRLab – an intelligent,operable,and low-risk mixed reality experiment platform,which not only strengthens the user’s experimental immersion experience in the mixed reality environment but also enables students to operate experimental objects with their hands.MRLab is an exploratory experimental platform with intelligent operation guidance and minimal operation rules,capable of establishing real-time interaction between real objects and virtual objects.The main innovation points of this study are mainly the following:(1)In this study,we designed a smart glove interaction device using a simple commercial sensor and a binocular camera.Devices such as indoor cameras or KINECT experience significant occlusion issues during virtual-reality fusion smart experiments,and they cannot detect small-scale experimental events and subtle movement gesture changes during experimentation.Hence,we developed the smart glove,which blends the binocular camera and all sensing devices into the glove to simplify the circuitry and improve safety.The smart glove can acquire multimodal information,including the user’s gesture signals and scene visual information,in a mixed reality environment and realize the sensing and fusion of this information.(2)In this study,we establish a Gesture-Voice-Visual-Symbol(GVVS)multimodal fusion algorithm based on gesture motion channel,speech channel,and scene visual channel.Traditional smart gloves accurately detect the user’s hand motion changes and subtle operations but have limitations in sensing the user’s behavioral intention.The GVVS algorithm adopts both feature layer fusion and decision layer fusion to process continuous or discrete data accurately and efficiently generated by users during virtual-real fusion experimental operations.It can quickly and accurately classify the user’s intention in experimental scenarios and guide the user’s performance based on their intention.(3)In this study,we designed MRLab,an experimental environment for intelligent experimental interaction in mixed reality scenarios.We established a channel for virtual-real interaction using a head-mounted device and demonstrated the intelligent experimental scenario under mixed reality conditions using a display.Meanwhile,according to the requirements of the secondary school experiment syllabus,we built interaction from the perspective of information acquisition,perception,recognition and cognition for the cognitive behavioral characteristics of secondary school students.Thus,users can operate experimental instruments and drugs in the real world and observe the corresponding experimental phenomena in the virtual world,satisfying their experiential immersion and providing a genuine operation experience.