Research On Multimodal Fashion Matching Recommendation Algorithm

With the advent of the Internet era,online shopping has gradually become one of the main ways of public consumption.Among them,garment sales occupy a large proportion in ecommerce platforms.Facing various kinds of clothing,how to recommend a set of both reasonable and beautiful clothing matching to users becomes a key problem.Previous research methods mainly focus on the interaction matching of fashion items and items or modeling items as a sequence,and although they achieve certain effects,they largely ignore the rich global correlation between items and the complex association between items.Two fashionable collocation frameworks for hierarchical masking and hierarchical graph convolution are proposed for the above problems,and the specific work mainly includes:(1)A fashion compatibility prediction framework based on a multi-layer masked Transformer has been proposed.This network architecture is built on hierarchical comparative relationships,and determines the compatibility of a complete outfit based on the complex features of individual items.The model is mainly composed of three parts: a visual semantic embedding module,a multi-layer masked module,and a fashion compatibility prediction module.Firstly,a Transformer encoder is introduced to extract the overall features of the complete outfit,making the feature extraction more comprehensive and thus obtaining more accurate compatibility judgments.Secondly,considering the complex relationships between clothing items,a hierarchical network is proposed to extract global features from low to high levels in the outfit.Finally,a new comparison method is introduced,which compares the global features of one item after being masked in sequence with the global features of another item after being masked,to determine the correlation between the two masked items and achieve clothing compatibility matching.(2)A fashion matching framework based on hierarchical fusion graph convolution has been proposed.The framework consists of five modules: visual feature extraction,semantic feature extraction,visual-semantic fusion embedding,hierarchical fusion embedding,and fashion compatibility prediction using hierarchical graph network.The aim is to inject complex relationships between items and global features into compatibility modeling.Firstly,low-tohigh level fusion features are extracted through a bimodal fusion autoencoder.Then,a multilevel embedding space is constructed with visual and semantic features,and the features of each item in the outfit are mapped to these layers of embedding space to build a hierarchical graph network.Finally,compatibility prediction is performed using graph convolution encoder and multi-layer perceptron,and an integrated learning approach is used to provide a compatibility score for the outfit,improving the performance of fashion matching.The proposed methods was validated on the Polyvore-T public dataset,and was compared to existing methods through experiments,showing improvements in both efficiency and accuracy.The first method achieved an accuracy of 91.8%,with a 1.34% improvement,while the second method achieved an accuracy improvement of 93%,and a 1.5% improvement in the fill-in-the-blank task.