| With the popularity of short video apps represented by ticktock and kwai,the video data stored on the network is also growing explosively.These video information often contains audio,image and other modal data.Correspondingly,traditional single modal retrieval has been unable to meet the daily needs of the application,people often hope to use one modal’s information to retrieve the information of other modals.However,the data structure of different modals are different,so how to strengthen the correlation between audio-visual modal data is a work worth further research.First of all,for cross-audiovisual matching and retrieval tasks,this paper proposes an isomorphic double-path network for feature extraction,so that the extracted features are as similar as possible,and the number of parameters and computation are as low as possible while maintaining the complexity of the model.On this basis,this paper conducts experiments and studies on cross-audiovisual media relevance mining from the following aspects:(1)A cross-audiovisual media matching and retrieval method combining category constraint and triple constraint is proposed.At present,most of the mainstream cross-audiovisual media retrieval and matching algorithms directly consider the correlation between different modal features,but this method will inevitably make the data features with different semantics lose their independence.At the same time,to describe different modal data can often be of the same semantic sharing of properties,according to this feature,this method through the study of the constraints of the shared properties and combined with considering the characteristics of relevance o f triplet constraint directly guide the model study shared expression,on the condition of no loss of semantic independence,effective response to the modal information retrieval and matching task.(2)A cross-audiovisual media association algorithm combin ed with selfsupervised learning is proposed.In real situations,the data often has the identity information only,if we want to get other attributes of information often requires a lot of human manual annotation,this method try to create the rest of the covariate,use an unsupervised learning to generate pseudo tag instead of the originally covariate function,improve the performance of the model,and prove the effectiveness of this method through experiments.(3)A cross-modal learning method based on generative adversarial network is proposed.In this method,by using adversarial learning mechanism to further strengthen the connection between the different modal data,and puts forward a new multi-modal center loss function at the same time strengthen th e modal and the correlation between modal,on the premise of unused category constraints ensures that the meaning of the modal data independence,reduce the amount of calculation,and in the model were improved.The above three cross-audiovisual media relevance mining algorithms proposed in this paper have carried out sufficient experiments on large-scale audio-visual dataset.Compared with the current mainstream cross-modal methods,their performance has been significantly improved and an in-depth study has been conducted in the field of cross audio-visual media. |
