The Research Of Machine Awareness And Bio-inspired Perception Based On Audio-Visual Information

The work accomplished in this thesis focuses on the research of intelligent perception approach to provide machine with artificial awareness ability.The proposed solution is based on the saliency principle to extract heterogeneous information,which to be fused to generate higher level knowledge of scene and event for perception.The contributions of this thesis are as follows: A)Investigation of the state of the art in saliency based visual and auditory information acquisition.B)Realization of a salient environmental sound detection approach based on the heterogeneous fusion process of multiple saliency features which obtained from temporal,spectral and visual domains respectively.C)A short-term Shannon entropy approach is proposed to estimate the global saliency of sound from the perspective of informatics.D)To better deal with overlapping sound and pseudo salient sounds,a computational inhibition of return(IOR)model is initially proposed to validate the temporal saliency detection result.E)Realization of a classification method based on the combination of MFCC feature and novel fuzzy representation of spectral energy distribution.F)Realization of a salient foreground object detection approach based on the objectness characteristic of foreground object.By using sparse representation,the objectness detection is based on the reconstruction error of image feature vector over a learned background dictionary.G)Realization of a classification approach by fusing visual saliency and objectness detection together in a human-like way.H)Proposition and realization of information probability model(IPM)of object and scene.The negative property is modeled based on the concept of structural saliency to accomplish complex perception task.The studies and experimental validations had confirmed that artificial awareness based intelligent perception ability can be obtained in complex environment for machine in an autonomous way.