Research On Key Techniques Of Tactile Signal Reconstruction For Cross-modal Communication

With the development of communication technology and the improvement of people’s experience requirements,the emerging multi-modal services,characterized as the integration of audio,visual,and haptic signals,will become the killer applications in 5G and beyond 5G era.In order to support multi-modal services,cross-modal communication emerged.However,when adopting crossmodal communications to haptic-dominant multi-modal services,there still face several tech-nical challenges.On the one hand,haptic signals are very sensitive to interference and easy to be damaged or even missing during transmission.On the other hand,it needs to generate virtual haptic signals when real touch sensory information is hard to be gathered.To get over the dilemma,this paper proposes a haptic signal reconstruction strategy for cross-modal communications.The main research content and innovations of this paper are as follows:First of all,according to the characteristics and transmission requirements of different modal streams,this paper constructs a cloud-edge collaboration-based cross-modal communication architecture for haptic-dominant services.The architecture uses device-to-device(D2D)technology to directly transmit haptic streams between terminals while retaining the existing audio-visualdominated multimedia architecture.Among them,the resource utilization is improved by multiplexing the cellular resources on the D2 D link which is responsible for the transmission of haptic stream.Moreover,this architecture supports haptic signal reconstruction at the receiver.Experimental results show that compared with other multi-modal transmission schemes,the modal stream transmission quality under the proposed scheme is better.Then,an audio-visual-aided haptic signal reconstruction(AVHR)approach is designed by leveraging the potential correlation among modalities,according to the damage and loss of haptic signals caused by transmission interference in communication.It can be further divided into three components: feature extraction by cloud-edge transfer,shared semantic learning by multi-modal fusion,and haptic signal generation by semantic constraints.Experiments on a standard audio-visualhaptic dataset and a practical cross-modal communication platform show that the proposed AVHR approach has better recon-struction performance when compared with the competing schemes.Finally,a fine-grained audio-visual-aided haptic signal reconstruction(FG-AVHR)approach is proposed to solve the problems of weak supervision and weak pairing in the existing multi-modal datasets.First,for the problem of weak supervision,a deep clustering algorithm based on cross-modal transfer is proposed for fine-grained classification of learning samples.Next,for the problem of weak pairing,the shared semantic learning by triplet constraint is proposed,which lays a foundation for the re-construction of haptic signals.Finally,haptic signal generation by clustering semantic constraints is performed to achieve fine-grained haptic signal reconstruction.Experimental results show that the FG-AVHR algorithm further improves the quality of reconstructed haptic signals.