Research On Multi-modal Learning For Imbalanced Modal Data

In real world applications, data are often with multiple modalities. Previous works assumed that each modality contains sufficient information for target and can be treated with equal importance. However, it is often that different modalities are of various importance in real tasks, e.g., the facial feature is weak modality and the fingerprint feature is strong modality in ID recognition. In this paper, we focus on improving the performance of weak modality with the auxiliary strong modal information, while reducing the expenses of strong modality. There get three major results in this paper as follows:First, we point out that different modalities should be treated with different strategies and propose the Auxiliary information Regularized Machine (ARM), which works by extracting the most discriminative feature subspace of weak modality while regularizing the strong modal predictor.Second, we propose a training strategy, ACQUEST (ACtive QUErying STrong modalities), which exploits strong modal information by actively querying the strong modal feature values of "selected" instances rather than their corresponding ground truths. An inverse prediction technique is also proposed to resolve the problem of combination explosion during feature querying, and make the ACQUEST a unified optimization form.Third, we propose a novel deep learning framework for the Imbalanced Modal problem. By connecting learned weak modal features with label information and strong modal features simultaneously, this model transfer the strong modal knowledge to weak modal domain by gradual dropping the connecting strong modal feature node, thereby we can only need weak modality at test time.