| With the development of artificial intelligence technologies such as deep learning,the demand for robots to achieve refined and intelligent perception of the external world is increasing.However,robots face various challenges in real-world environments,and one of the most important problems is how to explore the intrinsic connections and inherent properties of multimodal inputs such as vision,touch,and hearing,so as to effectively utilize multimodal information to perceive and understand the environment.To address this problem,this paper extensively explores and analyzes the correlation among different modalities,explores the common representations of different modalities,and constructs an algorithmic mapping from one or more modal spaces to another modal space based on generative adversarial networks,thereby obtaining cross-modal information representations that help robots perceive objects accurately.Specifically,the main research findings of this paper are as follows:(1)A U-Net network module based on a self-attention mechanism is constructed,which can effectively handle the characteristics of tactile spectrogram and visual texture data.Specifically,by integrating the self-attention module on the skip connections of the U-Net,the dependencies between the highly heterogeneous features of the tactile spectrogram and visual images can be effectively measured and the importance of the local and global visual-tactile information under more complex geometric constraints can be adaptively considered.(2)A generative adversarial network-based cross-modal generative model for visual haptics(ASVT-GAN)is constructed,which can effectively mine the correlation features of highly heterogeneous data in shared space,establish the projection mapping relationship between the source modality and the target modality,and generate highquality target modality data.Firstly,to reduce the probability of generating data that do not conform to a specific class or label and to improve the interpretability and controllability of the generated data,the model embeds the classification information of the pre-trained classifier and fuses it with the low-dimensional latent space features of the U-Net generation model based on the self-attention mechanism.Then,the lowdimensional feature vectors that incorporate semantic information and label information of the source modality are fused,upsampled,and supervised by the selfattention module to obtain high-dimensional representations of the target modality data.Finally,adversarial loss,feature matching loss,and pixel-level loss are introduced to update the model parameters in order to supervise the generator to generate realistic,clear,and more diverse target modal data.In this process,the discriminator employs spectral normalization technique to alleviate the instability problem during the training of the neural network and improve the generalization ability of the network.After various analytical experimental validations,it has been shown that the proposed ASVTGAN captures the nonlinear features of different modalities more deeply compared to directly using image-to-image translation models,thus improving the quality of image generation.(3)A cross-modal generative model based on the common representation of visual,auditory,and haptic modalities(CRCM-GAN)is constructed,which can learn the intramodality and inter-modality information variability and form a multimodal common representation space to achieve efficient modal transformation.Firstly,the spatial distribution of the different modal data is modeled to obtain low-dimensional representations of the different modalities.Then,cross-modal correlations are used to learn the common representation,and the tactile autoencoder network is used to retain the tactile reconstruction information to capture the semantic coherence of tactile itself.Finally,two category discriminators are used to constrain intra-modal highdimensional data and inter-modal low-dimensional feature constraints simultaneously.Through various analytical experimental validations,it has been shown that the proposed CRCM-GAN can adapt to complex constrained environments and has better generalization compared to the two modal generation tasks. |
