Reinforcement Learning Based Power Adapta-Tion And Spectrum Access In Cogmesh

Wireless communications are experiencing rapid growth around the world. Currently, the limited frequency band is overcrowded and there hardly exists space for the emerging wireless services. Cognitive radio (CR) is a promising radio possessing the intrinsic ca-pability to exploit the potential spectrum efficiency in time and space by sensing a wide range of authorized frequency and identifying the temporary unused spectrum holes. Cog-nitive wireless mesh networks, named as Cog Mesh, is featured as a self-organized and self-configured network architecture combining the CR technologies with distributed mesh structure. In this dissertation, we investigate the non-cooperative power control problem in underlay CogMesh as well as the opportunistic spectrum access problem in overlay CogMesh, and the contents are listed as follows:We first consider intercluster connection between the neighboring clusters under the framework of CogMesh. Specifically, we focus our emphasis on the case in which the two neighboring clusters are overlapping. Traditional intercluster connection usually needs four transmissions to complete the message exchange. By applying the idea of physical-layer network coding, which enables the gateway node to forward the signals received from the two clusterheads simultaneously, the number of transmissions reduces from4to2. More-over, we study the problem of how to choose the transmission power level at the gateway node under a stochastic networking environment where the primary users transmit with random power levels. Based on the reinforcement learning theory, we propose dynamic programming based power selection algorithm and (?)-greedy based intelligent intercluster connection algorithm. The main advantage of the two algorithms is that the gateway node makes optimal choice by interacting with the networking environment with incomplete information about the primary users.Then, we discuss the non-cooperative power allocation problem in CogMesh sce-nario. When all users transmit over the same channel, CR users need to adjust transmission power levels in order to guarantee the interference they cause to the primary users does not exceed a certain threshold. Due to the CR users'selfish and autonomous properties, the problem is modeled as stochastic game, based on which the single-agent Q-learning is ex-tended to a multi-agent context and a multi-agent Q-learning power adaptation algorithm is derived. However, the killing drawback of the obtained algorithm is that the CR users need the opponents'strategy information to update their Q-values. Therefore, a conjecture-based multi-agent Q-learning stochastic power adaptation algorithm is proposed to achieve the optimal strategies with only private and incomplete information. The algorithm provably converges given certain restrictions that arise during the learning process.We finally address the problem of the opportunistic spectrum access in Cog Mesh. Under the Overlay spectrum sharing scheme, CR users transmit over the licensed spec-trum bands only when the primary users are "sleeping". We assume that the primary users' random behavior patterns are known a priori, the spectrum access among CR users lo-cating in different clusters is not only a non-cooperative game, but a stochastic learning process. The learning CR users build beliefs with the goal of achieving reciprocity about how the competing users respond to their own strategy changes. We adopt the linear belief model, based on which a best-response learning based spectrum access method and a gra-dient ascent learning based spectrum access method are proposed. Further, the networking environment is proved to be stable when all CR users deploy the two proposed learning methods.