Research On User-Perception-Aware Cross-Modal Transmission Strategy

With the rapid development of wireless communication technologies,multi-modal services,which are characterized by mutual support of traditional audio-visual services and emerging haptic services,are gradually becoming the mainstream of multimedia services.Efficient heterogeneous streaming transmission is one of the key technologies to support multi-modal services.However,existing transmission strategies,which deliver and process each modality signal separately,cannot meet the diverse transmission requirements of heterogeneous streams.Besides,the huge data volume,inevitable signal distortion,and signal loss during the transmission also influence users’ immersive experiences.To overcome this dilemma,this paper discusses the optimization of heterogeneous streaming transmission in multi-modal service scenarios,and proposes a cross-modal transmission strategy based on the human perception mechanism,aiming to comprehensively improve end service quality and user immersive experience.The main contributions of this paper are shown as follows:Firstly,this paper proposes a user-perception-aware cross-modal transmission framework by taking full advantage of human masking effect and the potential correlations among different sensory modalities.It consists of an efficient data compression approach at the sender and high-quality signal inpainting and reconstruction methods at the receiver,which aims at comprehensively improving the transmission quality.In particular,in order to deal with the increasing data scale,we design a joint audio-haptic time-frequency masking compression approach by leveraging human masking effect for audio and haptic signals.It jointly makes full use of time masking `and frequency masking to remove and compress the imperceptible content of audio and haptic signals,efficiently eliminating the amount of audio-haptic streams to be delivered and relieving the pressure of network transmission.Secondly,due to the serious distortion and partial loss of visual signals caused by transmission noise,this paper proposes an attention-guided cross-modal visual signal inpainting scheme by deeply exploring the revelance among different sensory modalities.Based on the encoder-decoder network,it performs cross-modal inpainting of partially defective visual sensory signals through touch sensory.Firstly,the attention mechanism is introduced to accurately locate the visual defect area,and charaterzie their corresponding transfer features.Secondly,it combines semantic and supervision information to decrease the distance between the haptic feature and the transfer feature,and extract the content of visual defect area as much as possible.Finally,we make full use of the inter-pixel loss and adversarial loss to perform cross-modal inpainting the defected visual signals by the mined haptic features.Finally,due to the issues of delayed and lost visual signal caused by high synchronization requirements and network congestion during transmission,this paper designs an audio-haptic fused cross-modal visual reconstruction scheme through fully exploring the consistency and complementarity between non-visual sensory modalities.It exploits the semantic revelance between non-visual sensory modalities to reconstruct visual signal.In this scheme,low-level features extracted from the received audio and haptic signals are firstly fused based on common semantics.Then,an adversarial mechanism is introduced for the fused features to capture the latent relevance in real visual space.Finally,a hierarchical fine-grained representation structure and knowledge distillation technique are adopted to realize the desired visual signal reconstruction.