Event-Driven Image Quality Enhancement

High-quality visual signals are essential for humans to acquire and communicate information.Based on the energy-integrated imaging principle,traditional cameras can acquire highquality static texture-rich visual content.However,in high-speed scenes,traditional cameras inevitably suffer from motion blur due to camera shake and object motion,reducing imaging quality.Motion blur not only seriously affects the subjective visual perception experience of humans,but also directly interferes with the decision and execution of computer vision tasks.Therefore,the research on quality enhancement techniques for motion blurred images is of great significance.Due to the significant loss of motion information during the imaging process,this is a typical ill-posed problem,and it is very important to constrain its large solution space.A feasible solution is to utilize auxiliary sensors to provide supplementary information.The event camera,a visual sensor inspired by biological vision,subverts the traditional energy-integrated imaging mode in favor of an energy-differential imaging mode.The event camera asynchronously and independently captures the intensity change of each pixel and encodes it as an event stream.Compared with traditional cameras,event cameras have attractive properties: responding only to dynamically changing scenes,high temporal resolution,large dynamic range,and low latency.As a result,event cameras are able to capture high temporal resolution and fine-grained motion information,which can fundamentally alleviate the ill-posedness of blurred image quality enhancement.This paper focuses on the crossmodal complementary fusion of the conventional camera signals and event camera signals to reconstruct high-quality images without motion blur.At the same time,in order to determine whether the image quality enhancement results satisfy human subjective perceptual experience and meet the quality requirements of different visual tasks,the image quality assessment method is studied to monitor the quality enhancement results and provide feedback to optimize the quality enhancement algorithms.This paper analyzes and researches the above problems in depth,and the main research contents and innovations are as follows:(1)A motion deblurring algorithm based on cross-modal feature decomposition and recomposition is proposed.Aiming at the problem that existing methods only consider inter-modal complementarities(unique features)while ignoring inter-modal correlations(shared features),the proposed method is inspired by the divide-and-conquer strategy,which models the cross-modal fusion problem as a combination of modal-shared and modal-unique features.First,in order to improve the controllability and interpretability of the feature decomposition,correlation-driven constraints are designed to limit the solution space,i.e.,to reduce the correlation of inter-modal specific features,to increase the correlation of inter-modal shared features,as well as to promote the orthogonality of intra-modal shared and specific features,completing the feature decomposition? then,a bi-directional cross-attention based fusion strategy is proposed to propagate and exchange the properties of modal-shared and modal-specific features along the long range to accomplish feature recomposition,realizing cross-modal complementary fusion.The experimental results show that the proposed method can reconstruct high-quality images in the baseline blurry dataset as well as in real blurry scenes.(2)A motion deblurring algorithm based on cross-modal spatio-temporal calibrationaggregation is proposed.Aiming at the problem that existing methods do not explicitly perform cross-modal complementary learning,which leads to redundancy in spatial domain fusion,a mutual information minimization-driven cross-modal calibration strategy is proposed to explicitly suppress the redundancy to enhance the complementarity,and then a cross-modal synergistic attention mechanism is designed to balance the modal contributions to achieve cross-modal spatial domain collaboration? Aiming at the problem that existing methods lack the guidance of motion information in spatial-temporal modeling,which leads to the problem that neighbor blurring information will interfere with spatio-temporal feature fusion,the event-guided spatiotemporal attention confidence is proposed to emphasize more reliable spatiotemporal features,and then a cross-temporal coordinate attention module is designed to model the channel and position dependence of the spatiotemporal neighboring features with the current spatial-domain features to achieve cross-modal temporal domain collaboration.Experimental results show that the proposed method is able to reconstruct high-quality video sequences in the baseline blurry dataset as well as in real blurry scenes.(3)A no-reference image quality evaluation model based on hierarchical semantic quality degradation is proposed.Aiming at the situation that the existing methods cannot meet the diverse quality requirements under different cognitions,the proposed method first takes the different quality requirements of visual information in different cognitive scenarios as the entry point to construct the semantic quality metrics based on the local detail clarity,regional contour clarity,and global conceptual fidelity.Then,inspired by the mechanism of extracting different features from different regions of the visual cortex,the proposed method employs a multi-task specific model to extract the hierarchical features with interpretability.Next,inspired by the Bayesian brain theory,the relationship between the hierarchical features is systematically derived based on joint probability,which guides the decoupling and fusion of hierarchical features to complete the coarse-grained image quality assessment.Furthermore,a bidirectional degradation attraction strategy is designed to recurrently learn degradation dependencies between hierarchical semantics to realize the fine-grained hierarchical semantic quality assessment.Finally,the proposed quality enhancement algorithm and quality assessment algorithm are combined to build a high-speed motion deblurring system.