Research On Action Recognition Methods Based On Infrared And Visible Spectrum

Action recognition based on infrared and visible spectrum aims to identify action occurring in multimodal video through computer vision and machine learning methods.With richer information provided,multi-spectral action recognition can be widely used in many occasions,such as airports,stations,schools and other public places with large traffic,and can also be applied to places where there is a need for night monitoring such as banks and warehouses.In this thesis,the advantages and complementarities of infrared and visible spectrum are studied.The detailed research work is as follows:Firstly,a new multi-spectral action recognition dataset is expanded and proposed based on an existing infrared and RGB action dataset.This new dataset increases the number of samples,expands the scene,angle,occlusion,interference and seasonal changes.During the capturing,the cameras of both spectra started simultaneously.Since there exists some differences in resolution and viewing angle between the two cameras,a registration method for infrared and RGB images is proposed.Aiming at reducing the difference between two spectra,a two-stage network is designed.This network first performs a mapping on the visible image,generating a mapped spectrum which is similar to the infrared image.In the second stage,the network utilizes the optical flow network and a sampler to transform the infrared image based on the mapped spectrum to achieve crossspectral registration.Inspired by the attention mechanism of the human eyes in video analysis,a learningbased adaptive pooling method is proposed for multi-spectrum action recognition.Conventional pooling methods treat different videos and frames identically.However,for different actions,people's focus should be various.Therefore,this method incorporates the pooling process and action classification as an optimization problem.Through the training process of action recognition,the optimal pooling parameters of each action categories can be automatically learned.Compared with the traditional pooling methods,the adaptive pooling method achieves better performance on both low-level and high-level features and improves the accuracy of action recognition.Finally,an action recognition method for uncertain spectrum is studied.Extensive research shows complementarity between infrared and RGB spectra.However,in practical applications,sometimes only one spectrum can be used.For example,only infrared spectrum can be used in places demanding privacy protection.Therefore,a multi-spectrum based generative adversarial network is proposed.Through supervised learning,the network utilizes generators to produce a full modal representation containing both infrared and RGB information using only one spectrum.Simultaneously,action recognition is achieved by an added branch of the discriminator.Experimental results reveal that the proposed method improves the accuracy of action recognition of single spectrum.