Research On Multi-modal Learning In Imbalanced Modal Environment

As data collection ability improves,we can obtain more and more multi-modal data.In many real applications,there are imbalanced modal phenomena:different modalities show different importance to specific task,one modality is called strong modality when it has strong correlation with task,and weak modality is less correlated with task.In imbalanced modal environment,different modalities have different precision and need different extraction costs.Based on these observations,methods proposed in this paper are as follows:1.This paper proposes a thorough model reuse scheme named Fixed Model Reused(FMR).FMR uses strong modality(fixed modality)and trained model(fixed model)to guide the training process of weak modality,which helps the model of weak modality to be more discriminative.FMR uses both labels and fixed modality/model which obtained from other task to supervise the training process.Besides,it uses"Knockdown" to reduce the connection between network and fixed model.In test phrase,it only needs weak modality to make a reasonable prediction.Fixed model reuse is helpful to reduce time cost,data amount,and expertise required.2.This paper proposes a novel end-to-end serialized adaptive decision approach named Discriminative Modal Pursuit(DMP),which can automatically extract instance-specifically discriminative modal sequence for reducing the cost of feature extraction in test phase.DMP uses LSTM to extract modality.In each step,if current modality predicts well,then DMP chooses a similar modality to ensure predicting,if it predict badly,then DMP chooses a modality which is totally different from current modality to rich the modal type to help predicting.When the confidence of predict or the cost meets thresholds,DMP stops modality extraction.By making use of the correction between modalities fully,DMP could reduce modality extraction cost effectively.