Controllable Image Editing Based On Deep Neural Networks

Image editing covers a wide range of image processing tasks,which generally involves processes of changing the content of images at the pixel level.Most methods that are based on deep neural networks are deterministic models.An issue with deterministic models is that they usually only deal with a certain degree of editing,and the whole mapping process is uncontrollable.Changes in real life are typically uncertain.For example,facial aging is a gradual process.The users would want to obtain facial images at different ages(controllable outputs),instead of only at a certain age.Another example is that in the image denoising task where different images contain different noise levels,the users would hope that the network could deal with various noise levels self-adapatively(controllable inputs),instead of only being able to deal with a certain noise level.Based on the hypothesis that the controlling variables that correspond to the degree of editing of the network are implicitly embedded in the network's parameter set and can be extracted and utilized,in this paper,we focus on controllable image editing based on deep neural networks.Specifically,we take the degree of editing of the network as the main line,and carry out research on the problem formulations and model frameworks at three levels from specific degree of editing,discrete intermediate degrees of editing to continuous intermediate degrees of editing,successively.1.Aiming at the problem of image editing with a specific editing degree being uncontrollable,this paper reveals the relationship between the dimensionality reduction and representation ability of the building block in the image editing network of a specific degree of editing,and the relationship between the stackability of the building block and the controllability of image editing.This paper discovers that the stackable characteristics of a single-layer auto-encoder include its dimensionality reduction characteristics — being capable of capturing the dimensionality reduction patterns that are more consistent with the spatial structure of data,detecting the repeated structures in data,and the hidden layer being able to learn features that are important for the vision tasks;and the relationship between the number of hidden layer nodes of the auto-encoder and the intrinsic dimensionality— better performance is achieved when the number of hidden layer nodes is set as the intrinsic dimensionality of the input.2.Aiming at the problems which involve supervised,discrete processes of change,the gradual editing algorithm based on stacked generalized auto-encoders is proposed and the image editing framework with discretely controllable outputs is constructed,thus achieving discrete control over the output.By introducing supervision for discrete intermediate degrees of editing,it makes it easier for the network to achieve the transition from the local optimal solution to the global optimal solution.The supervision not only helps achieve discrete control of the output,but also helps better complete the original task.The characteristics and stackability of the auto-encoder make it more suitable for handling the discretely controllable image editing problem.3.Aiming at the problems which involve weak-supervised,continuous processes of change,the network modulation algorithm based on adaptive instance normalization is proposed and the image editing framework with continuously controllable outputs is constructed,thus achieving continuous control over the output.By taking advantage of the implicit correlation priori among different degrees of editing,implicit supervision for continuous intermediate editing states is established in order to achieve continuous modulation of the degree of image editing and increase the controllability of the model on the output end.The complex parameter representation learned by convolutional neural networks and its flexibility make it more suitable for handling the continuously controllable image editing problem.4.Aiming at the problem of the network not being able to edit the input as needed,the input-adaptive network modulation algorithm based on spatial feature transformation is proposed and the image editing framework with continuously controllable inputs is constructed,thus achieving continuous control over the input.By constraining and establishing more uniform supervision for intermediate editing states,the network is able to learn the mapping from the input to the degree of editing it needs,which further increases the self-adaptability of the model on the input end.Eventually,continuous and self-adaptive image editing are achieved simultaneously,which have controllable outputs and inputs,separately.Through the above-mentioned study,the methods that are proposed in this paper cover various cases in the controllable image editing problem,from single to multiple degrees of editing,from discrete to continuous intermediate states,from full-supervision to non-supervision of the intermediate states.The goal of image editing is to learn the mapping from one image domain to another.While the goal of controllable image editing is to not only achieve the transformation between the two image domains but also figure out the direction of the mapping.Being subject to deep learning's sensitivity to the training data,its advantages can not be fully reflected with limited samples.The controllable image editing that is proposed in this paper is able to learn the transformation between two image domains and establish the process of gradually transforming from one image domain to another,in the case of limited or lack of samples,thus reducing the dependency on the training data and achieving continuous domain adaptation.The application in a variety of visual tasks has proved the practical value of the proposed methods in this paper and the general applicability in image editing problems.