Real-time Human Intrusion Detection Using Audio-visual Fusion

Public security has received more and more attention in recent years.With the rapid development of society, traditional manual surveillance cannot meet the safety-monitoring demands of current society. The nextgeneration surveillance system will be intelligent surveillance systems.With the development of Internet of Things and information fusiontechnology, multi-sensor joint surveillance has become one researchhotspot. If one surveillance system only use single sensor in the complexscenarios, it may exist information blink areas, which will result in highfalse alarm rate. Multi-sensor information fusion can overcomeone-sidedness and limitation of the single sensor. In addition, multi-sensorinformation fusion can improve robustness and decline false alarm rate bythe complementation of multi-sensor information.Bayesian network is used to fuse audio-visual information, and then weuse the Bayesian network model to detect human intrusion. A multi-sensorfusion surveillance system is designed and implemented for the powerstation surveillance.First, as the huge computation of HOG human detection make itimpossible to be used to real-time image processing. By extracting theregion of moving objects, we limit HOG feature computation to theextracted region. One spatial-temporal joint detection region shrinkingmethod is developed to reduce the computational load. Our proposalmethod can be used for real-time surveillance.Second, we use MFCC+GMM method to detect footstep. Consideringthe characteristic of footstep and synchronization, we use speakeridentification scheme but make some changes in feature extraction preprocessing step.Third, since the recognition accuracy of HOG-based human detectionwill drop markedly under occlusion, considering the fact that humanintrusion must be accompanied with footsteps, footstep recognition is usedto improve the detection robustness. One Bayesian Network is developedto fuse audio and video signals at decision level to detect human intrusionevent.At last, a multi-sensor fusion surveillance system is designed andimplemented to monitor power station.