| Applications arising from computer, telecommunications, and manufacturing systems lead to many challenging problems in the simulation, stability, control, and design of stochastic models of networks. The networks are usually too complex to be analyzed directly and thus one seeks suitable approximate models. One class of such approximations are diffusion models that can be rigorously justified when networks are operating in heavy traffic, i.e., when the network capacity is roughly balanced with network load.;We study stochastic networks with time varying arrival and service rates and routing structure. Time variations are governed, in addition to the state of the system, by two independent finite state Markov processes X˜ and Y˜. Transition times of X˜ are significantly smaller than the typical interarrival and processing times whereas the reverse is true for the Markov process Y˜. We first establish a diffusion approximation for such multiscale queueing networks in heavy traffic. The result shows that, under appropriate heavy traffic conditions, properly normalized queue length processes converge weakly to a Markov modulated reflected diffusion process. More precisely, the limit process is a reflected diffusion with drift and diffusion coefficients that are functions of the state process, the invariant distribution of X˜ and a finite state Markov process which is independent of the driving Brownian motion. We then study the stability properties of such Markov modulated reflected diffusion processes and establish positive recurrence and geometric ergodicity properties under suitable stability conditions. As consequences, we obtain results on the moment generating function of the invariant probability measure, uniform in time moment estimates and functional central limit results for such processes. We also study relationship between invariant measures of the Markov modulated constrained diffusion processes and that of the underlying queueing network. It is shown that, under suitable heavy traffic and stability conditions, the invariant probability measure of the queueing process converges to that of the corresponding Markov modulated reflected diffusion.;The last part of this dissertation focuses on ergodic control problems for discrete time controlled Markov chains with a locally compact state space and a compact action space under suitable stability, irreducibility and Feller continuity conditions. We introduce a flexible family of controls, called action time sharing (ATS) policies, associated with a given continuous stationary Markov control. It is shown that the long term average cost for such a control policy, for a broad range of one stage cost functions, is the same as that for the associated stationary Markov policy. Through examples we illustrate the use of such ATS policies for parameter estimation and adaptive control problems. |
