| Depth images and hyperspectral images provide important information sources for downstream tasks in different tasks in the computer vision.They can be widely used in multiple fields such as target detection,tracking,recognition,and three-dimensional reconstruction.However,obtaining depth images and hyperspectral images at the same time is a quite challenging task.Existing methods often require two independent systems to obtain these two types of image information,and then calibrate,align,and merge their results.This not only increases the burden of hardware equipment and cost,but also may lead to inaccurate results.In recent years,with the continuous development of deep learning technology,deep learning-based methods have reduced the cost of obtaining depth images and hyperspectral images.However,most researches deal with depth images or hyperspectral images separately,and there are few studies on simultaneously obtaining depth images and hyperspectral images from a single-frame RGB image.To solve this problem,this paper proposes a multi-task learning method that simultaneously recovers depth images and hyperspectral images of scenes from single-frame RGB images,achieving an integrated processing of obtaining depth images and hyperspectral images.The main work includes:(1)A depth estimation method based on defocus blur and spectral focus stack is proposed,breaking through the limitations of traditional defocus depth estimation methods,and having significant advantages.Compared with other defocus depth estimation methods,our method performs better on the FlyingThings3D synthetic dataset,while also showing good performance on the HS-D dataset after fine-tuning.This innovative depth estimation method combines the advantages of defocus blur and spectral focus stack to estimate scene depth accurately and quickly with good generalization ability.The proposal of this method has important practical application value for solving the problems existing in traditional depth estimation methods.(2)A multi-stage spectral-wise Transformer based on Spectralwise multi-head self-attention mechanism is used for hyperspectral image representation learning to achieve high-accuracy hyperspectral reconstruction,which solves the problem of simultaneously obtaining hyperspectral and depth map data of a unified scene for joint training of two tasks.Compared with traditional methods,the multi-stage spectralwise Transformer used in this paper utilizes a spectral attention mechanism to better model the similarity between spectra and reduces computational complexity by expanding the receptive field of the network.Experimental results show that on the FlyingThings3D synthetic dataset,the multi-stage spectral-wise Transformer used in this paper outperforms other hyperspectral reconstruction networks and provides strong support for subsequent joint recovery of depth-spectral information.(3)A multi-task learning network is proposed aiming at simultaneously recovering scene’s depth information and spectral information from single-frame RGB image.This method combines multistage spectral-wise Transformer with dual-decoder U-Net fully utilizing correlation between depth estimation and hyperspectral reconstruction using all-focus RGB image for training.Through experimental verification on FlyingThings3D dataset our proposed method achieves simultaneous recovery of depth image and hyperspectral image from single-frame defocused RGB image demonstrating superior joint recovery ability for depth-spectral information. |
