Event-Based Image Reconstruction And Quality Evaluation

Efficient and high-quality visual information capture is very important to promote the development of artificial intelligence technology.Although traditional imaging devices can obtain high-spatial resolution images with rich textures,there would be problems such as blur,underexposure or overexposure in some scenes like high-speed motion and high-dynamicrange(HDR).In addition,with the increases of frame rate,traditional cameras will generate a large amount of redundant data,which greatly occupies storage resources and computing resources.As a bio-inspired imaging system,the imaging principle of event camera,which senses the change of light intensity based on energy difference,is completely different from the traditional imaging system which relies on the light intensity integral sampling principle.The event camera only responds to the dynamically changing scene content,and each pixel of it works independently.When the light intensity on a pixel changes and the change reaches a threshold,the pixel generates an event signal.Compared with traditional cameras,event cameras have the advantages of fast imaging speed,high temporal resolution,high dynamic range,less data redundancy,and low power consumption.Nevertheless,the output of event camera is a series of asynchronous and sparse event streams,which is not intuitive for view,nor the application of existing image frame-based computer vision algorithms on event camera.Thus we need to convert the events into the format of traditional images/videos.At the same time,in order to ensure the reconstructed image could provide comfortable visual experience for human observer,we additionally need evaluate the image quality.Image quality assessment not only monitors the quality of the reconstructed image,but also serves as an optimization index and feedback to improve the reconstruction algorithm.In view of the above requirements,this paper carries out a series of event camera-oriented researches,include event-based image reconstruction and image quality evaluation.The main research contents and innovations are as follows:1.A novel unsupervised event-to-image reconstruction(E2IR)method based on multi-scale domain adaptation(DA)is proposed.Due to event signal contains sparse texture and the ground truth is hard to collect,existing methods could not recontructed high-quality images effectively.Thus,a full-time event tensor containing rich background information is firstly constructed based on the long and short event completion strategy to alleviate the problem of sparse texture of event signal.Next,based on the correlation between E2 IR and traditional image enhancement(IE),i.e.,the two task both aim to restore high quality textures from degraded information,we futher propose an unsupervised DA based image reconstrction method.The proposed DA method adopts adversarial trainging on multi-scale output space to learn domain-invariant features,and consequently transfers the knowledge in image enhancement to E2 IR.Experimental results show that our proposed Multi-scale Domain Adaptation based E2IR(MDAE2I)method can reconstruct high quality images effectively.2.A no reference(NR)image quality assessment(IQA)method based on hierarchical quality analysis is proposed.Existing NRIQA network lacks big data and can’t represent the quality degradation accurately and comprehensively.To solve these problems,we first establish a large scale dataset with more one billion images,and we compute a quality score for each image based on merging multiple full reference(FR)IQAs.Next,inspired by the hierarchical process of human visual system(HVS),we design a deep cascade convolutional neural network(CNN)based on hierarchical feature integration.Driven by big data and the end-to-end hierarchical-joint optimization,the hierarchical degradation is measured.Experiment results show our method could comprehensively analyze the degradation of hierarchical features,and achieve satisfactory performance.3.A novel NRIQA model with active inference is proposed.To alleviate the dependence of NRIQA on the reference image,inspired by the internal generative mechanism(IGM),we firstly propose an active inference module based on generative adversarial network(GAN)to simulate the IGM process,and predict the primary content of a distorted image.During training,the model is optimized by two IGM-inspired terms: the semantic similarity constraint for semantic consistency and structure unsimilarity constraint for structure completeness.Next,based on correlation between distorted image and its primary content,a multi-stream CNN is designed to analyze the image quality from content-/distortion-/degradation-dependent perspectives.Experimental results demonstrate that our proposed method could predict the primary content accurately,thereby realizing accurate IQA and achieving perfect generalization ability and robustness.