CNN-based Hyperspectral Reconstruction From RGB Videos

Hyperspectral imaging technology collects image or video data in dozens or hun-dreds of adj acent narrow spectral bands,containing rich spatial and spectral information.By calibrating and analyzing the hyperspectral data,we can derive a continuous spec-trum curve for every image pixel in the scene,which is useful in diagnostic minerals,object detection,medical science,environment monitoring,and military tracking.Since hyperspectral imaging has to obtain a large number of narrow-band spectral images simultaneously,real-time acquisition of hyperspectral data with high spatial res-olution is a tough research problem.As RGB videos can be viewed as the integration of hyperspectral videos in the spectrum,RGB videos and hyperspectral videos share pretty similar spatial textures and relatively strong spectral correlations.Besides,it is easy for RGB cameras to capture RGB videos with high spatial resolution and high frame rate.Therefore,reconstructing hyperspectral videos from RGB videos is a the-oretically feasible and practical scheme.Note that in industry RGB videos are always quantized and compressed with unknown quantization parameters for the purpose of reducing storage and transmission costs.Because there is a large difference in video quality between videos compressed by different quantization parameters,analyzing the quantization parameters of the compressed RGB videos and reconstructing hyperspec-tral videos accordingly is the key to our research.This dissertation revolves around using deep-learning-based approaches to recon-struct hyperspectral videos from RGB videos.And it can be roughly divided into two parts:spectral reconstruction from uncompressed RGB videos and from compressed RGB videos.We learn the RGB-to-hyperspectral mapping by exploiting the spatial and spectral similarities between RGB videos and hyperspectral videos,as well as the inter-frame redundancies.Specifically,the main contributions and the novelty of this dissertation are:1.Converting the problem of spectral reconstruction from uncompressed RGB videos to the spectral super-resolution of uncompressed RGB videos.First,we build a multi-scale image pyramid for motion compensation and align the RGB frames,then the aligned frames are fed to the residual spectral reconstruction network for inter-frame information fusion and spectral super-resolution.The output of the spectral reconstruction network is the reconstructed hyperspectral video.Compared with other state-of-the-art methods,the proposed method can significantly improve the objective and subjective quality of the reconstruction results,and the time cost is relatively small.2.In terms of the spectral reconstruction from compressed RGB videos with un-known quantization parameters,we first propose a novel quantization parameter estimation module to estimate the quantization parameter of the video,then we adopt a feature-pyramid-based motion compensation network to align the RGB frames.After that,the quantization parameter information and the aligned frames serve as input to a single-model architecture for hyperspectral reconstruction,where videos of different qualities undergo adaptive spectral super-resolution pro-cedures,thus costing relatively small computational and storage resources.To the best of our knowledge,it is the first time that compressed RGB videos are utilized for hyperspectral reconstruction and preliminary experimental results on the test set demonstrate the superiority of our method over the state-of-the-art methods on various different quantization parameters.