On Markov Decision Processes In Wireless Communications

The evolution of wireless communication systems exhibits the characteristics ofcomplexity, diversity, intellectualization, persistence, and microminiaturization.Wireless communication systems not only undertake speech transmission tasks, but alsoimplement missions such as monitoring, controlling, and decision-making. Theapplication demands motivate researchers to go further in the field of control theory inwireless communication. Markov decision process (MDP), which becomes thecutting-edge research in the filed of methodology of wireless communication, is one ofthe effective solutions for the controlling and decision-making problems in wirelesscommunication. In this dissertation, aiming at the challenging of MDP in wirelesscommunications, the issues in this area, such as inaccurate model problems, parametersuncertainty, and optimal solutions, are systematically investigated.In this dissertation, the research methods are as follows. First, the issue of thedelayed system state is investiaged via observations and analyzing. According to thefinite MDP theory, the MDP-based power control approaches based on the delayedsystem states are studied. Based on the robust dynamic programming, the robust powercontrol methods under channel uncertainty are investigated. In addition, the differenttime-scale issue is studied via the multiple-level MDP theory. Finally, the large-scaleMDP problems are investigated via the system state aggregation theory. The maincontributions in this dissertation can be concluded as follows.(1) On the delayed system state issues in wireless communications.Due to the overhead during the spectrum sensing in cognitive radios, currentspectrum status is not available for cognitive users. The wireless communicationsystems make decision based on the previous spectrum status. Therefore, the reasonwhy the spectrum sensing delay occurs is investiaged. For the Rayleigh fading channel,a discrete-time Markov chain model is proposed, which can accurately characterize boththe fading characteristics and the primary users’ behaviors. Extensive simulations showthe degradation of the delayed system states on the performances of the power controlapproaches, and the proposed model is more accurate than existing models. (2) On the power control approaches based on the delayed system states.For the interference management of cognitive radios and protection of primaryuser’s transmission, a channel outage probability constraint is proposed. Based on thedelayed MDP theory, the power control approaches based on the delayed system statewith channel outage constraint are proposed. The optimal power control policy isderived via dynamic programming. Furthermore, aiming at the complexity issue ofoptimal policy, the hybrid power control approach is proposed. Extensive simulationsshow that the hybrid approach can largely reduce the complexity and remain reasonableperformance.(3) On the robust power control approaches under channel uncertainty.The channel model parameters are estimated based on the limited measurementdata, and the estimation errors exit. However, the MDP-based approaches are sensitiveto the errors of the model parameters. Therefore, the impacts of uncertainty on theMDP-based approaches are investigated. Two different uncertainty models, i.e., theinterval matrix model and the likelihood model, are proposed. According to the robustdynamic programming, the robust power control policies and the correspondingalgorithms for the inner problems are proposed. Simulation results show that theproposed approaches outperform traditional methods.(4) On the joint energy scheduling and transmission control approaches.The wireless communication systems equipped with energy harvesting devices areconsidered. The joint energy scheduling and transmission control polices in suchsystems are investiaged. First, the characteristics of the energy harvesting process forthe current issue of model errors are studied. Then a novel non-homogenous Markovchain model is proposed. This model can not only track the main trend of the energyharvesting process, but also describe the random factors. A framework based on themultiple-level MDP is proposed for the different time-scale issue. This framework canmodel the energy scheduling and transmission control problems. The hybrid valueiteration and dynamic programming approach is proposed, and the optimal energyscheduling policy is obtained. Simulation results show that the proposed approachesoutperform the existing methods.(5) On the fast algorithms for the large-scale MDP problems in wirelesscommunications. Aiming at the high complexity issues in the large-scale MDP problems, thefactored expression for the large-scale MDP is proposed. According to the factoredMDP, the fast algorithm based on the system state aggregation theory is proposed. Theproposed algorithm shows low complexity and reseaonable good performancecompared to the optimal solution.