Multimodal Image Enhancement Based On Orthogonal Meta-Space

Image enhancement technique adaptively assigns appropriate aesthetic features to the given ordinary images(mainly manifested as disharmony of illumination,contrast,etc.)while preserving the details of the image.Given a normal image,most existing work only outputs a unique enhanced image with a uniform aesthetic style that comes from a given set of reference images.However,the aesthetic preferences of different users and specific applications of the technique may have large differences,which requires the image enhancement method to have the ability of multi-modal processing using a single model,giving a set of diversified output for the user to choose.At the same time,image content and aesthetic characteristics need to be fully disentangled and multi-modally fused.In order to realize multi-modality enhanced output,this paper proposes a multimodal image enhancement framework based on orthogonal meta-space,extracts the aesthetic characteristics of visual attractiveness in the reference image,and explicitly encodes it into an orthogonal meta-space.Specifically,this paper first uses the encodedecoder and adversarial training strategy to extract the characteristic and content features of the high aesthetic quality image.Next,the style coding of the reference image is mapped into a characteristic meta-space formed by a set of orthogonal bases,where the orthogonal bases are implemented by introducing orthogonal regularization losses.In the test phase,given any ordinary image,the feature of the content is extracted by the encoder;at the same time,multiple characteristic features are randomly sampled in the style meta-space;finally,the original image content encoding and the characteristic feature are respectively combined and sent to the decoding.The resulting multimodal enhanced image is obtained.In addition,in order to improve the stability of characteristic,content feature disentanglement and fusion in multi-modal scenes,this paper improves the adaptive instance standardization module for image enhancement tasks.The module normalizes the content code using the channel-dimensional statistical information of the image characteristic code and its position information in the meta-space during the training process,so that the decoder simultaneously obtains the high-frequency style features and the content features of the ordinary image.Besides,we propose to maximize the mutual information between the index code of meta-space and the prediction of the discriminator,which enhances the modeling capacity of the space.The improved adaptive instance mormalization module and the mutual information optimization enhance the stability of the characteristic-content feature disentanglement of same image,and cross-image characteristic-content feature fusion.In the experimental part,the method has achieved favorable performance than other related works on various quantitative metrics based on aesthetic score,inception score and diversity score.In addition,the visualization of the enhanced images and the user study results also show that the method can generate beautiful and diverse images.