Semi-Markov Switching State-Space Control Processes And Its Applications

With the progress of information science and technology, the numerous network communication systems are proliferated around the world. Promoted by increasing service demands, the network communication systems are becoming more functional mightiness, structural complexity. While multiple control strategies are employed to interact with stochastic environment frequently, the dynamics of network communication systems are more complex. Performance analysis and optimization play an important role in the design and operation of network communication systems, and have crucial effects on operation efficiency, service ability and QoS guarantee. There are various approaches such as perturbation analysis in systems and control, Markov decision processes in operation research, reinforce learning in computer science address the optimization of stochastic dynamic systems. The main challenge is that the existing approaches are insufficient to deal with the optimization of complicated network communication systems. The most important issue in related research fields is that how to characterize and utilize the system special features efficaciously, and by which to develop novel optimization approaches for solving the existing key technology problems in real-life network communication systems.Motivated by the optimization of modern network communication systems, an analysis and optimization framework called semi-Markov switching state-space control processes (SMSSCPs) is introduced. This framework possesses hierarchical dynamic structure and adopts event-driven control policy. The modeling, performance analysis, and event-based optimization of the proposed processes are discussed. With the definition and classification of events, the state space is divided into multiple layers to characterize the hierarchy of system dynamics, which makes the modeling more flexible and scalable. By adopting event-driven policy, the policy space is reduced, and the computation of associated optimization algorithms can be saved considerably. The feature of event-driven policy and the structure of dynamic hierarchy are exploited to release the dependence of optimization algorithms on the knowledge of systems parameters, and thus improve the adaptability of optimization algorithms. The proposed approach is employed to address some key technology problems in network communication systems, such as adaptive bandwidth allocation in wireless multimedia communication networks, policy optimization of dynamic power management, and modeling and optimization for networked media service systems.The analytical model of SMSSCPs is formally introduced by characterizing different layer events, constructing the semi-Markov kernel and semi-infinitesimal generator for the system under control of event-driven policy, and defining the performance and switching cost functions as well as performance measurement. Based on the definition of semi-Markov potential, the Poisson equation is derived for the switching processes, and the performance sensitivity formulas, performance gradient and performance difference, are constructed. For the optimization of deterministic policy, by exploiting the information of dynamic hierarchy contained in semi-Markov kernel or semi-infinitesimal generator, the caparison theorem of event-driven policy is derived. On the basis of this theorem, the dependence of policy iteration on the knowledge of system parameters is released, and the adaptability is obtained. By utilizing the feature of event-driven policy, the applicability condition of policy iteration, i.e. independence-action assumption, is relaxed, and thus the policy iteration can sufficiently handle the optimization problem formulated as SMSSCPs. The potentials are aggregated according to events, and thus the numbers of potentials needed to be calculated are reduced. An online policy iteration algorithm is presented, and its convergence is proved. For the optimization of random policy, on the basis of performance gradient formula, the gradient estimate of average performance with respect to event-driven switching policy based on a single sample path is derived. Combined with stochastic approximation, a policy gradient-based online adaptive optimization algorithm is proposed. By exploiting the feature of event-driven policy, the computation is reduced and the adaptability is improved. On the benefit of hierarchical dynamic structure of SMSSCPs, it is proved that this algorithm can converge to the global optimum with probability 1.The issue of adaptive bandwidth allocation in wireless multimedia communication networks is considered firstly. An event-driven stochastic analytical model is introduced to formulate the adaptive bandwidth allocation problem as a constrained optimization problem. In this framework, the adaptive bandwidth allocation and call admission control schemes are combined. The separation between incoming traffic for each class and the high priority of handoff calls over new calls are taken into account. New call blocking probability, handoff dropping probability, and average allocated bandwidth are considered as QoS constrains. An online optimization algorithm that combines policy gradient estimate and stochastic approximation is proposed to handle this constrained problem. This algorithm doesn't depend on the prior knowledge of systems parameters, and is well adaptive to various application environments. Simula- tion results demonstrate that the proposed algorithm is efficient to maximize network revenue while QoS constraints guaranteed.Policy optimization of dynamic power management (DPM) is then discussed. For the optimization of stochastic and timeout policies for DPM, event-driven semi-Markov switching models are presented to characterize the hierarchical dynamics of power-managed systems embedded in stochastic environments. These models accurately capture the system dynamics during the transitions between operation states or lasting at idle state waiting for the timeout value expired. The modeling accuracy ensures the reliability of analysis and the effectiveness of optimization. Two online adaptive optimization algorithms are proposed for addressing the optimization of stochastic and timeout policies respectively. By utilizing the feature of event-driven policy and the structure of dynamic hierarchy, the proposed algorithms are adaptive, with less computational cost and power efficient. By analyzing the steady-state behaviors of power-managed systems, the equivalence on power-performance tradeoff of timeout and stochastic policies is revealed, and the equivalent relation between these two types of policy is derived.The third application issue is modeling and optimization of networked media service systems. The networked media service systems driven by multi-layer control mechanism are modeled as a three-layer SMSSCP. This model provides a combined analytical framework for policy optimization of adaptive resource allocation, dynamic service composition, and access requests assignment. Streaming media server cluster is the fundamental component of networked media service systems. A dynamic file grouping (DFG) strategy is presented for load balancing in such clusters. This dynamic strategy is based on a two-layer SMSSCP model. It effectively improves the system availability by balancing the workloads among delivery servers within the cluster and increasing the access hit ratio of cached files in delivery servers to mitigate the limitation of I/O bandwidth of storage node. An online policy iteration algorithm is employed to optimize the DFG policy. This algorithm exploits the feature of event-driven policy to alleviate the dependence on the exact knowledge of system parameters such as user access patterns and reduce the computation, which makes it more efficient and feasible in practical applications.