Design And Development Of Dangerous Behavior Detection System Based On Multimodal Information Fusion

Nowadays,China’s public safety problems emerge in endlessly.Dangerous behavior detection can effectively reduce the incidence of accidents in public places and is of great significance to maintaining social harmony and stability.Using intelligent monitoring technology to monitor the possible dangerous behaviors in hospitals,schools and other public places can timely find and deal with potential safety hazards,prevent the occurrence of safety accidents,and protect the safety of people’s lives and property.As the carrier of multimodal information,video contains many kinds of modal information,such as RGB map,optical flow map and bone map.However,in real applications,the video scene is complex and changeable,and most behavior detection algorithms often detect for single mode,which is difficult to fully express the video behavior information,and there are some problems such as low efficiency and insufficiency of feature extraction,which affect the detection effect.Therefore,this paper improves the two stream network and spatio-temporal convolution network,fully mining the characteristic information of different modes in video,proposes a multimodal information fusion model,and designs a dangerous behavior detection system based on this model.The main research work is as follows:(1)Aiming at the problem of low utilization of the two isomorphic modal feature information of RGB graph and optical flow graph,the dual flow network is improved.Firstly,the video data is preprocessed to obtain the RGB image and optical flow image in the video;Secondly,the dense connection network is introduced to increase the network depth and realize the feature level fusion between the two modes,so as to improve the feature utilization;Finally,in order to eliminate the adverse effects such as the increase of computational complexity caused by the increase of network depth,the learning packet convolution is embedded to optimize the network structure,which is conducive to improving the efficiency of network training.Experiments on the kinetics-600 dataset and NTU RGB+D dataset show that the improved method is effective.(2)Aiming at the problem that RGB modal behavior detection algorithm is easily disturbed by complex environment and can not fully express human behavior in video,bone graph is added as the third mode to supplement video behavior information.Openpose is used to extract bone images,and STN and Tam attention modules are used to improve the spatio-temporal convolution network to improve the feature extraction ability of the model.The improved two networks are combined to form the final multimodal information fusion model,which not only has the feature fusion between RGB graph and optical flow graph,but also realizes the complementarity of the three modal information through decision-making level fusion.The experimental results show that the performance of the model is superior.(3)The multi-modal information fusion model is further trained through the self-built dangerous behavior data set,and the dangerous behavior detection system is developed by using Python language to build the system environment,pyqt5 to design the system interface and My SQL database.The detection of three common dangerous behaviors in public places such as falling,fighting and climbing over the guardrail is realized,and the development of the whole system is completed.