Research On Visual Relationship Understanding And Image Generation Towards Fashion

With the continuous improvement of people’s aesthetic ability,the needs towards decent garments have become significant increased.However,in a sense,not every person has a great taste towards aesthetics,automatic clothing matching has thus become an urgent real-world demand.Meanwhile,in reality,the compatible clothing items are manually devised by fashion designers,which is both time and labor consuming.To this end,intelligent fashion design deserves our close attention.Last but not least,people usually need to check the try-on look of the interested fashion items to determine whether to buy it or not.Nevertheless,the online fashion shopping cannot provide the physical try-on service due to the spatial constraint.Accordingly,we should also pay attention to the virtual try-on study.In light of this,we mainly aim to devise a series of methods to meet the real-world demands in fashion domain.i.e.,how to understand the visual compatibility relationship of an outfit,how to generate matching fashion item images and how to generate virtual try-on images.Towards this end,we pay attention to the research on the visual relationship understanding and image generation towards fashion domain.Although existing methods have achieved great success,they still perform far from satisfactory due to the following limitations.1)Current visual compatibility understanding studies only consider the collocation compatibility modeling,while neglecting modeling the compatibility of the outfit try-on effects.2)Existing compatibility generation methods mainly adopt the one-to-one manner.However,the matching items have the one-to-many compatibility relationship.Facing the complicated scenario,traditional one-to-one mapping cannot globally exploit the compatible design modes and fail to generate disverse items.3)The traditional try-on image generation methods only aim to synthesis the single-view try-on result,that is,keeping the person’s pose unchanged while simply changing the clothing item,which cannot meet people’s real-world try-on demand.To address the aforementioned limitations,we carry out the research work in the following three aspects:(1)Mutual-enhanced Multi-View Compatibility Relationship UnderstandingWe focus on evaluating the compatibility from both collocation and try-on views and we propose the Collocation and Try-On Network(CTO-Net)for FCM.In particular,considering that visual compatibility among discrete items in an outfit can be affected by multiple latent factors,we devise disentangled graph learning for collocation compatibility modeling(CCM)to uncover fine-grained compatibility.To utilize the limited try-on images to fulfill the try-on compatibility modeling(TCM),we devise the integrated distillation learning scheme.In addition,we employ the mutual learning strategy to encourage both CCM and TCM to transfer knowledge from each other.Extensive experiments on the real-world dataset FOTOS demonstrate that our model significantly outperforms the state-of-the-art methods.(2)One-to-Many Visual Compatibility Relationship-based Item GenerationIn this paper,we explore the one-to-many visual compatibility relationship among fashion items and we propose the Distribution-based Matching Fashion Item Design(DMFID).DMFID explores the compatible design modes for the query item and generates diverse matching items.In particular,we decompose the complex task into two stages:diverse compatible shape generation and diverse compatible item generation.Meanwhile,to explore the complicated oneto-many compatibility relationship,we utilize all the ground truth compatible items to learn a compatible distribution for each query item,which can globally capture the latent compatible design modes.Extensive experiments on the real-world dataset IQON3000 demonstrate that our model outperforms the state-of-the-art methods.(3)2D Spatial Alignment-based Virtual Try-On Image GenerationWe introduce a new try-on setting,which enables the changes of both the clothing item and the person’s pose,and we present a Spatial Alignment-based Virtual Try-on Network(SAVTN).We decompose the try-on task into two stages:clothing deformation and try-on image generation.Firstly,we propose a shape enhanced clothing deformation scheme to accurately warp the given new clothing item by learning the spatial relationship between the clothing item and the person context.In particular,a target body shape mask prediction module is introduced to provide the intermediate guidance.Then we present an attentive bidirectional generative tryon network to synthesize the realistic try-on image,which simultaneously regularizes the attentive clothing-person alignment and the bidirectional generation consistency.Moreover,we create a large-scale FashionTryOn dataset from the e-commercial website Zalando②.Extensive experiments validate the superiority of our model.