| Multimodal services including audio,video,and haptic information are expected to enhance users’ immersive experience,and are becoming killer applications in the 5G or even 6G era.In order to support multi-modal business,cross-modal communication technology emerges as the times require.However,when haptic information and visual information are equally important in multimodal business,the design of cross-modal communication schemes still faces many technical challenges.On the one hand,limited available transmission resources cannot simultaneously accommodate visual streams and haptic streams with different transmission requirements,resulting in inevitable competition between audio-visual and haptic modalities.On the other hand,the quality of the received visual and haptic signals cannot be guaranteed due to the degradation and delay of the transmission conditions.To flexibly deal with these possible problems,an adaptive cross-modal transfer strategy is proposed in this paper.It includes a visual-haptic mutual aid streaming protocol at the sending end and a visual-haptic signal mutual aid reconstruction method at the receiving end.For the former,inspired by the correlation of human visual-haptic perception,the transmission redundancy of visual stream and haptic stream can be eliminated.For the latter,by exploring semantic correlations,damaged or delayed visual or haptic signals can be reconstructed with the help of another modality.In particular,the primary focus and novel contributions of this study can be outlined as follows:First of all,a cross-modal communication architecture for visual-haptic equally important businesses is proposed.It includes a transmission compression mechanism at the sending end and a signal reconstruction model at the receiving end,that is,a flexible transmission scheduling scheme and a cross-modal reconstruction scheme are jointly utilized to adaptively adjust the transmission strategy.The architecture realizes redundancy elimination of visual stream and haptic stream to improve resource utilization,and at the same time ensures that the packet loss and delay problems of visual and haptic signals are adaptively solved in communication.Secondly,a visual-haptic signal mutual aid reconstruction algorithm(VHMR)is designed,which makes full use of the semantic association of different modalities to realize mutual generation of visual and haptic data.Put the model on the receiving end and use it to generate missing and delayed visual or haptic signals.VHMR consists of four modules: feature extraction based on knowledge transfer,semantic association based on modality fusion,intra-modal signal reconstruction based on adversarial learning,and inter-modal signal reconstruction based on cycle consistency constraints.The outcomes of experiments conducted on a widely-accepted multimodal dataset,as well as on an actual cross-modal communication platform,demonstrate that the visual-tactile reconstruction performance of VHMR is impressive.Finally,in order to further refine the generation effect of the reconstruction signal,a constraint framework on hierarchical details is introduced,and a fine-grained visual-haptic mutual reconstruction algorithm(HFG-VHMR)based on hierarchical semantic constraints is designed.First of all,knowledge transfer not only considers the differences in semantic information within modals between domains,but also considers the information differences between modalities,which can effectively improve the effect of feature extraction;then,through modal fusion,potential semantic associations are mined;finally,based on Hierarchical semantic constraints enable cross-modal signal reconstruction,and use the classification understanding model as the supervision and reference of the reconstruction network to effectively guide high-quality,fine-grained signal reconstruction.Experimental results show that the HFG-VHMR method further improves the quality of visual-haptic mutual aid reconstruction. |
PDF Full Text Request