Research On Optical Flow Registration Algorithm For SRGB And Bayer Pattern Image

Image registration,the task of aligning two or multiple images with relative displacement,is widely used in industrial fields such as smart phones,digital cameras,autonomous driving,and aerospace remote sensing fields such as satellite imagery.Current image registration al-gorithms can be divided into models based on feature points matching and homography matri-ces estimation,variational optical flow registration algorithms and learning-based optical flow registration algorithms.Optical flow describes the relative displacement between image pix-els.Compared with transformation-matrix-based algorithms,optical-flow-based algorithms can align images pixel by pixel with higher accuracy.On the basis of summarizing the advantages and disadvantages of existing algorithms,this paper focuses on designing high-performance and high-precision optical-flow-based image registration algorithms from three perspectives:data,model,and post-processing.In terms of data,this paper is committed to building large-scale,high-quality optical flow image registration training and testing datasets.SRGB(standard RGB)domain image with 3 channels of red,green and blue is the most widely used image form.To build a large-scale,multi-scene sRGB domain registration dataset,object labels of existing semantic segmentation datasets are collected to generate the foreground objects in the image sequences to be regis-tered,Internet and real-shot images of various types and scenes are collected to generate the backgrounds of the image sequences.The optical flow labels are obtained by adding transfor-mation matrices to the foreground and background objects independently and randomly and recording the pixel displacements between images.The dataset constructed contains 1000+types of foreground objects,the richness of foreground categories and background scenes is significantly better than the existing datasets.Single-channel Bayer pattern raw field image,which consist of red,green and blue pixels arranged in a cycle of 2 by 2,is the most commonly used original image form collected by digital cameras.To solve the current lack of Bayer pat-tern image registration dataset,this paper integrates the ISP(digital camera image processing pipeline)simulation,noise modeling and motion simulation to transform the previously con-structed sRGB domain dataset into the current largest Bayer pattern dataset.In the aspect of model,we integrates the concepts of the feature pyramid,the relative dis-placement compensation using optical flow warping,the construction of cost volume and the multi-scale optical flow refinement to construct the two-frame basic model for the sRGB do-main image registration task.For Bayer pattern images,the basic pipeline of the Bayer pattern registration,which consists of input image arrangement,optical flow estimation,Bayer pat-tern optical flow warping and ghost detection in the warping results,is built.On this basis,the two-frame basic model in the sRGB domain is transformed into four basic models in the Bayer domain.To utilize the multi-frame information more efficiently,the split-attention mod-ule and the dot-product-attention module are designed.They can adaptively estimate the fusion weights to fuse the features extracted from the additional frame into the main optical flow net-work.Based on the above attention modules,we propose a split-attention multi-frame image alignment network,which can register multiple images in one inference process,and all align-ment results use the multi-frame information.In the Bayer domian,we design an optical flow layout which consists of 8 channels and is divided into 4 groups,called Bayer optical flow.Then we propose a repeatable correlation searching block guided by Bayer optical flow to search for similar features among the images to be aligned.Finally,the recurrent neural network is used to simulate the iterative optimization process of variational optical flow algorithms,the optical flow is optimized on a single resolution scale,and the number of iterations can be adjusted.In the aspect of post-processing,this paper analyzes the causes of ghosting effect in the im-age occlusion areas and the optical flow distortion areas.The ghost detection algorithms based on repetition areas detection in the optical flow backward warping results and overlapped areas detection in the optical flow foward warping results are designed according to the definition of the ghost.They can detect ghosts while warping images with optical flow without increasing the time complexity of the registration algorithms.This paper verifies the effectiveness of the large-scale sRGB and the Bayer pattern train-ing set we constructed.A series of experiments and robustness tests show that the basic optical flow registration models we designed can accurately align the sRGB and Bayer pattern images pixel by pixel.Compared with the current mainstream two-frame models,the multi-frame reg-istration model proposed can align multiple images more accurately,robustly and efficiently.The proposed bayer pattern recurrent correlation searching network achieves excellent results in objective evaluation metrics,subjective detail evaluation and multi-scene test.The proposed two ghost detection algorithms are also superior to other compared classical algorithms.