Statistical Texture Modelling Via Deep Neural Network

Texture refers to images that contain only random or repeating structures,which are ubiquitous in nature.The study of texture phenomena is not only the key to understanding human vision,but also has application value in computer vision and graphics.In recent decades,researchers have developed statistical tools for texture modeling.By using statistical models to extract statistical features in textures,researchers can efficiently characterize visual features of textures.Recently,deep neural networks have been introduced in texture modeling.Thanks to the strong expressive power of neural networks,such deep texture models have achieved modeling power greater than of that of traditional models.However,there are still some unresolved issues.First,because these deep texture models rely on neural networks pre-trained on large-scale datasets,they are not only incapable of modeling textures in domains where large-scale datasets are difficult to obtain(such as sound or video),but are also susceptible to pre-training data.In addition,since such models use complicated deep features,it is difficult to interpolate multiple models to obtain mixed textures with merged input visual features.Furthermore,such models can only deal with images that contain only a single type of texture with no background areas,i.e.homogeneous texture images,but cannot model general natural images.These problems greatly limit the application scope of texture models.In order to solve the above problems,this paper conducts the following research.To overcome the dependence on pre-trained neural networks,this paper proposed a new texture model as a generalization of existing deep texture models: by combining deep texture features with an energy-based generative model,the proposed model is capable of adaptive learning the network weights for each input texture independently,thus no longer relying on pre-trained neural networks.Therefore,the model can not only applies to any type of texture modeling tasks,including image,video and sound texture modeling,but also can avoid the influence of pre-training data and generate higher quality textures.Extensive experiments were conducted in this paper to show that the model outperforms the baseline methods on texture synthesis,expansion,and inpainting.In addition,this model is the first texture model that can perform the expansion of video textures,or the expansion and inpainting of sound textures.To interpolate deep texture models,this paper proposed a mixing algorithm based on stationary Gaussian model.This paper showed that the second-order statistics in the existing deep texture models can be represented by a unified stationary Gaussian model,so that the interpolation of these deep statistics can be calculated by interpolating stationary Gaussian models.Since no iterative process is required,the mixing algorithm adds only negligible time to the texture modeling process.Further,when used for style transfer tasks,the algorithm can mix different styles to generate images with mixed styles.This paper experimentally showed that,compared with baseline methods,the proposed algorithm not only produces smooth transformations between textures or styles,but also preserves the fine texture structure in the input image.In order to extend the applicability of texture modeling to natural images,this paper proposed a texture extraction algorithm that can model textural elements contained in natural images.Specifically,by considering image features as a set of patch features,the algorithm formulated the texture extraction problem to a partial distribution matching(PDM)problem.In order to solve the PDM problem,this paper used the partial Wasserstein-1 distance(PW distance)to describe the similarity between probability distributions,and developed the corresponding optimal transport theory: it proved that the PW distance has an equivalent Kantorovich-Rubinstein(KR)dual form.Further,this paper explored the properties of the optimal solution of the KR dual form,and the proved relationship between the various equivalent forms.Based on the above conclusions,this paper proposed a Partial Adversarial Network(PWAN)model for the large-scale PDM problem,and applied it to the partial alignment task of feature sets,resulting in a texture extraction algorithm.This paper experimentally showed that PWAN can efficiently solve large-scale PDM problems,and can efficiently model textures contained in natural images.