Research On Robot Cross-Modal Perception Technology Based On Visual-Auditory Cues

Robot perception technology is one of the most important method of robot intelligent development. Nowadays the advances of robot vision have not only greatly broadened the application of the robot but also improved the efficiency of the robot's work. But in most conditions the performance of the robot does not always satisfy the demands of the owner. Especially in some complex and dynamic environment, robot is needed to be adapted to the circumstance rapidly and can make decisions according to the changes of the circumstance and the task. Visual sensor alone can not fulfill these real-time requests. Human hopes that they can break through their cognitive confines with the help of the robot, at the same time, scientists try to make the robots more human.Recent years, with the developments of cognitive neuroscience and anatomy, human has understood more of the brain. The cross-modal perception and attention mechanism are investigated deeply and widely. Plenty of results have been gained. The traditional AI seems to be trapped after a long period of progressing, people naturally turn their eye to the study of brain and cognitive science which will bring breakthrough in robot investigation. Human applies attention mechanism to cull information selectively, and analyzing visual or other sensory data purposefully in order to answer special queries posed by the observer. The cross-modal perception also help people to explore and cognize the environment. The amount of information provided by the sensors typically exceeds the processing capacity of the system. If human wants to configure the limited computation resource to the most emergency and concerned task of the robot, cross-modal perception and attention mechanism is a best way to achieve the aim. Therefor, there is important academic significance and applied value in the research of this method. This paper has set up a model of cross-modal perception and attention mechanism of robot, thus make the robot detect the object more effectively and efficiently, so the robot can take further cognitive activity.The main content of the research and its results are as follows:1. The correlative fruits of brain and cognitive neuroscience, neurophysiology in human cross-modal perception is analyzed and summarized. Human's auditory pathway, visual pathway, characters of the auditory neuron receptive field are also summarized.2. According the inspiration of the results of investigation in neurobiology, this research simulates human's visual perception space and auditory perception space, a model of robot auditory space perception map that transformed into a type of image is constructed, so the image processing technology can be used in auditory cues processing. This can lead to a more robust and accurate object detection.3. A robot visual space perception map is constructed based on visual cues and image processing technology. The map can be registered with the auditory space perception map. So a cross-modal space perception map is set up. On this map, robot orientating attention model is established, which can make the robot detect the visual or(and) auditory object more rapidly.4. Design a emulational experiment system of object orientating attention, then classical detection events is validated based on LabVIEW, a graphic language dealing with the auditory and visual cues.Mass results of the experiments indicate that the model of auditory orientating attention in this research has a precision of -5o to 5o in object detection. Auditory object detection has nearly the same precision with that of visual object detection. At the same time, it also complement the visual object detection when the object is blurred, partly blurred, sheltered, or placed in a dark surrounding in which the object is invisible.