End-to-End Model-Driven Deep Neural Network For Physical Layer Transceiver Design

The stringent performance requirements of 6th generation(6G)systems,such as ultra-high data rates,extremely high reliability and low latency,are spurring worldwide studies on defining the next-generation multi-user multiple-input multiple-output(MU-MIMO)systems with advanced transceivers that can integrate far-reaching applications ranging from autonomous systems to extended reality.In 6G systems,advanced physical layer transceiver design for next-generation MIMO systems is a core technology that comprises innovative designs in transmit and receive processing architecture,hybrid precoding,beam selection,channel estimation,channel feedback,and semantic communications,etc.To satisfy the requirements of 6G,iterative optimization algorithms with domain knowledge are an attempt to design the transceiver.Although these algorithms have exhibited satisfactory system performance,they typically have high computational complexity and make the deployment difficult.Recently,with the development of artificial intelligence(AI),deep learning techniques provide a solution due to the strong learning ability and the superiority of dealing with big data.To improve its interpretability and generalization ability,researches further propose a novel deep learning technique,named model-driven deep-unfolding network.It fully exploits the domain knowledge and combines the optimization algorithms with deep neural networks(DNNs).This thesis aims to propose efficient model-driven networks for different modules in physical layer transceiver,and incorporates them into end-to-end deep learning to effectively solve the problems in communication signal processing.First,based on the characteristics of physical layer transceiver design,this thesis proposes a framework for model-driven networks in matrix form.Then,we implement the proposed framework to solve the sum-rate maximization problem for precoding design in MU-MIMO systems.An efficient network architecture is developed based on the classic weighted minimum meansquare error(WMMSE)algorithm.Specifically,the iterative WMMSE algorithm is unfolded into a layer-wise structure,where a number of trainable parameters are introduced to replace the highcomplexity operations.To train the network,a generalized chain rule is proposed to depict the recurrence relation of gradients between two adjacent layers in the back propagation.Simulation results show that the proposed IAIDNN efficiently achieves the performance of the iterative WMMSE algorithm with reduced computational complexity.Second,this thesis develops a framework of deep deterministic policy gradient(DDPG)-based model-driven network with adaptive depth for different inputs.Specifically,the optimization variables,trainable parameters,and architecture of deep-unfolding NN are designed as the state,action,and state transition of DDPG,respectively.Then,this framework is employed to deal with the channel estimation problem in massive MU-MIMO systems.Specifically,we unfold the sparse Bayesian learning(SBL)-based algorithm into a layer-wise structure and employ the proposed DDPG-based framework to solve this channel estimation problem.To realize adaptive depth,we design the halting score to indicate when to stop.Simulation results show that the proposed algorithm outperforms the conventional optimization algorithms and DNNs with fixed depth with much reduced number of layers.Next,this thesis investigates the joint design of beam selection and digital precoding matrices for MU-MIMO systems with discrete lens array(DLA).The investigated non-convex problem with discrete variables and coupled constraints is challenging to solve and an efficient framework of joint neural network(NN)design is proposed to tackle it.Specifically,the proposed framework consists of a deep reinforcement learning(DRL)-based NN and a deep-unfolding NN,which are employed to optimize the beam selection and digital precoding matrices,respectively.As for the DRL-based NN,we formulate the beam selection problem as a Markov decision process(MDP)and a double deep Q-network(DDQN)is developed to solve it.Regarding the design of the digital precoding matrix,we develop an iterative WMMSE algorithm induced model-driven NN,which unfolds this algorithm into a layer-wise structure with introduced trainable parameters.Simulation results verify that this jointly trained NN remarkably outperforms the existing iterative algorithms with reduced complexity and stronger robustness.Furthermore,this thesis proposes an end-to-end deep learning-based joint transceiver design algorithm for massive MIMO systems,which consists of DNN-aided pilot training,channel feedback,and hybrid analog-digital precoding.To further reduce the signaling overhead and channel state information(CSI)mismatch caused by the transmission delay,a two-timescale DNN composed of a long-term DNN and a short-term DNN is developed.In particular,the analog precoders are designed by the long-term DNN based on the estimated high-dimensional full CSI matrices and updated once in a frame.In contrast,the digital precoders are optimized by the short-term DNN based on the estimated low-dimensional equivalent CSI matrices and updated once in a time slot.Simulation results show that our proposed technique significantly outperforms conventional schemes in terms of bit-error rate performance with reduced signaling overhead and shorter pilot sequences.Finally,this thesis designs a framework for the robust end-to-end semantic communication systems for image transmission to combat the semantic noise.In particular,we analyze sampledependent and sample-independent semantic noise.To combat the semantic noise,the adversarial training with weight perturbation is developed to incorporate the samples with semantic noise in the training dataset.Then,we design the masked vector quantized-variational autoencoder(VQ-VAE)with the noise-related masking strategy.We use a discrete codebook shared by the transmitter and the receiver for encoded feature representation,where a constellation has been designed for efficient transmission.To further improve the system robustness,we develop a feature importance module to suppress the noise-related and task-unrelated features.Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.The first two works propose framework of model-driven network for transceiver modules in physical layer communication with satisfactory performance and low computational complexity,while the latter three works incorporate model-driven networks to end-to-end learning framework.They jointly provide some theoretieal basis and effeetive teehnical solutions for the further evolution of 5G to the 6G.