Fine-grained Clothing Image Retrieval Based On Multimodal Features

When browsing clothing products on e-commerce websites or shopping software,people often have diversified retrieval needs,such as the need to retrieve clothing with certain similar clothing attributes,and the need to retrieve clothing with only one different clothing attributes,that is,clothing attribute operation,These requirements are fine-grained clothing image retrieval,which can not be well met by the current search engines.In view of the above retrieval requirements,it is necessary to extract the features accurately related to the specific attributes of the image and clothing.The development of deep learning makes this operation possible.Guided by deep learning theory,this paper delves into the fine-grained clothing image retrieval.The main work and innovations are summarized as follows:（1）A clothing attribute tree is constructed according to the clothing attributes in two real clothing datasets,and a hierarchical attribute embedding（HAE）method for clothing attribute representation is proposed ground on the attribute tree.This method can effectively incorporate the hierarchical relationship of clothing attributes,and ensure that when it is used as the representation of clothing attributes,two child node attributes under the same parent node attribute can share the characteristics of its parent node attribute.（2）A network HAEN based on hierarchical attribute embedding is proposed,which is used for learning clothing similarity.The network takes images and attributes as input,learns the embedded subspace of multiple specific attributes,measures the fine-grained similarity in the corresponding subspace,and can effectively support the query of similar clothing images with given attributes.In the network,hierarchical attribute embedding is used as the representation of clothing attributes.For image features,the global features of clothing images are output through Resnet50,and then the hierarchical attributes embedding and the obtained global features are fed into the attention module of the model,so as to get image features which is relevant to specific attributes.After that,the image features related to the attribute are input into the embedding branch module,embedded into the subspace corresponding to the attribute,and the mask module is used to select the feature dimension related to the specific attribute.Finally,The result of dot product on the vector output by the network is the similarity between garment images.（3）For the more complex clothing image retrieval task of attribute operation,three different solutions are proposed based on feature replacement based on HAEN.They are the method based on the mean feature vector of clothing attributes,the method based on the reconstruction of feature vectors of clothing attributes and the method of adaptive combination of mean feature vector and reconstructed feature vector.Among them,the mean feature vector adopts the idea of replacing the individual with the whole,and is obtained by calculating the mean value of all the feature vector of clothing images in the retrieval database,The reconstruction feature vector is obtained by inputting the query image’s own information and attribute operation semantics into the reconstruction module.Finally,the combination of the two features is used.The paper explores the impact of the three attribute feature vectors on the retrieval performance.（4）Plenty of experiments have been conducted on two large-scale real clothing datasets.The experiments show that the network HAEN proposed in this paper has good performance when calculating the similarity between clothing images according the clothing attribute,which is superior to the existing state-of-the-art methods.In addition,for attribute operation,the three methods proposed in this paper are better than the benchmark model,and have good performance on the task of attribute operation clothing image retrieval.