Human-machine Hybrid Intelligence Oriented Image Quality Assessment And Application

At present,artificial intelligence technology has achieved rapid development and makes a profound impact on intelligent applications in various fields.However,artifi-cial intelligence algorithms are inherently uncertain,which makes them have potential risks and problems in intelligent deployment.In order to solve this raised problem,an important trend is to develop human-machine hybrid intelligence,introducing human cognitive models into artificial intelligence systems,finding the optimal combination of the perceptual abilities of humans and the computational capabilities of artificial intel-ligence,and promoting better solving of practical problems.Before being handed over to humans or machines for intelligent tasks,the original image data need to be collected,compressed,transmitted,and reconstructed.During this process,the image quality is affected,resulting in lost information in varying degree,which will not only bring about the uncertainty of machine recognition,but also have a bad influence on human recognition,causing people to make inaccurate judgment.Therefore,it is very important to accurately quantify and evaluate the quality of the image data for human-machine hybrid intelligence.Accurate quality evaluation on the one hand can guide the entire processing flow of image optimization,such as image coding,on the other hand,it can also guide the design of machine learning algorithms or human-machine hybrid intelligent algorithms.This dissertation focuses on several specific recognition tasks in surveillance sce-narios,analyzes the impact of image distortion on human recognition and machine recognition,and explores image quality assessment algorithms and applications in human-machine hybrid intelligent scenarios.The content of the dissertation includes the fol-lowing four parts:(1)Gradient-based machine uncertainty estimation scheme.Machine uncertainty reflects how the image distortion affects the recognition performance of machine learn-ing algorithm,which can also be regarded as the image quality for machine recognition.Our idea is inspired by the works of neural network interpretability which show a strong correlation between the back-propagated gradients and the network's response with the input sample.We insist on that retrospecting the back-propagated gradients also has potential benefits on identifying sample distributions.Therefore,we propose a novel out-of-distribution detection scheme to detect out-of-distribution examples.In detail,we introduce a sample-specific perturbed loss function to generate discernible gradi-ents,and compute the norm of gradients of the loss function w.r.t the input image as detection score.Moreover,since back-propagated gradients may decay rapidly due to the suppression between positive and negative gradients,we propose a gradient clip-ping strategy to mitigate the inhibitory effect.Experimental results demonstrate that our method consistently outperforms the baseline method ODIN by a large margin,es-tablishing a new state-of-the-art performance in training-free approaches.(2)Image semantic quality assessment scheme in the surveillance scene.In this section,we focus on the impact of image quality on human recognition.Since this task has not been investigated,we build an image semantic quality assessment dataset and proposed a no-reference image semantic quality assessment scheme to evaluate the quality of image semantic information under surveillance scenarios.In the dataset,we aim at two representative foreground surveillance targets,namely,pedestrians and vehi-cles,extracting human faces and pedestrians for pedestrian targets and license plates for vehicle targets.We add two commonly used compression distortions JPEG and BPG,and motion blur,a common distortion in surveillance scenes.We then invite volun-teers to view the images,and finally collect the recognition results as semantic quality assessment scores for each distorted image.Based on the image semantic quality as-sessment dataset,we design a no-reference image semantic quality estimation network,and compared with traditional image quality evaluation algorithms.We further ana-lyze the differences of the perception of image distortion between machine and human,including the recognition accuracy comparison and gradient visualization analysis.(3)Human-machine co-judgment application based on image semantic quality as-sessment scheme.We build a new human-machine co-judgment framework based on machine uncertainty and image semantic quality assessment model.Compare with tra-ditional rejective learning scheme,our method not only considers the uncertainty of the machine,but also considers the potential biases and weaknesses of human recognition.Our method can combine the perceptual ability of human and the recognition capability of machines in more efficient way.We carry out experimental verification in the recog-nition task of the surveillance scene.The experimental results show that our method improves the recognition accuracy by more than 8%compared with traditional rejective learning scheme,and it also improves the efficiency of human decision-making partic-ipation.(4)The exploration of the application of semantic information in several image or video processing tasks.In this part,we explore the application of semantic information in several image or video processing tasks.We propose a collaborative training strategy based on semantic perception distance for image encryption that maximizes the differ-ence in semantic distribution.We propose an interleaved zooming block that can obtain global semantic structure information for image inpainting task.We propose a sequen-tial mesh flow image inpainting network that can obtain temporal motion information for video inpainting.In the experiments of the three tasks,we have verified that design of the algorithm with perceptual semantics,semantic structure,motion information,etc.can bring good gains to the performance.