Soft Segmentation For Sky/Cloud Image Manipulation And Retrieval

Cloud image understanding can be widely used in such areas like weather prediction and solar power generation.In editing images,the cloud is one of the most common elements.In natural scenes,the sky is a frequent image content.However,the sky in the image is often monotonous and have no clouds decorated.At that time,we can retrieve satisfied cloud images on the Internet,and fuse them into the original sky images to enhance the visual effect of the sky images.This paper aims at above problem about image editing and divides it into three questions as cloud image segmentation,cloud image retrieval and cloud image manipulation.Aiming at the segmentation of cloud images,this paper designs a soft segmentation method to get the whole appearance of clouds and local opacity.Cloud has no fixed shape and its local opacity is different and dynamic duo to its mobility,which makes the cloud difficult to represent.Therefore,this paper use a feature fusion strategy to represent the cloud,which takes full advantage of the powerful high-level semantic cues generated by a Deep Neural Network(DNN)and the classical low-level features from previous researches.The paper uses a Gaussian Mixture Model(GMM)to characterize the feature distribution of the cloud.Then,this paper designs a novel energy minimization formulation to soft segmentation,which maintain spatial and features homogeneity among soft segments while preserving discontinuities by assigning appropriate soft labels corresponding to their respective feature distribution.The paper uses an alternating optimization(AO)algorithm to solve above optimization problem and outputs the segments with soft labels.Finally,since the soft segments have no object labels,a pre-trained model is used to group the decomposed segments into a cloud object based on the Maximum Likelihood Estimation(MLE).Experiments verify that our method can obtain finer segments compared with existing methods.Being specific to the problem of retrieval various cloud images,this paper designs a method based on graph traversal to retrieval the cloud images,which queries by example(QBE).This paper uses the deep features extracted by the DNN to grade similarity and re-rank the images with an exploring method,and outputs the cloud image sets that rank higher.Through the method in this work,we can get multiple retrieval results for cloud images.According to the cloud manipulation,the paper tries several manipulation processes based on the above segmentation and retrieval algorithm,such as the measurement of opacity of clouds,clouds recoloring,clouds removaling and clouds clipping.