Reinforcement Learning Based Resource Allocation Strategies For Cloud Brokers

Cloud computing has achieved a significant success in the industry.More and more firms and personal users have been using cloud computing services over Internet,which contributes to the development of the cloud computing.As the demands of cloud users become more and more complicated,it's difficult for them to purchase appropriate service from service providers.There exist cloud brokers,who make profits by buying cloud resource from the providers and subleasing them to the users.However,due to limited budget,the broker in the cloud market can only get limited capacity of resource from cloud providers,moreover,the cloud user demands are usually stochastic in the cloud computing market.Therefore,in order to make more profits in a finite period of time,the broker needs to allocate the resource to the more “valuable” users.Based on this,in this thesis,we analyze the resource allocation strategies of cloud brokers and focus on consider the resource allocation strategies in non-competitive environment and competitive environment.The research of this thesis is listed as follows:(1)Firstly,we get the basic settings of the issue we analyze in the thesis.We first model the users' stochastic demands,which means that the number of cloud users entering the market at each stage is dynamically changing and the demand of each cloud user is dynamically changing as well.Then,we model the users' choice model over multiple cloud brokers in the competitive environment.Finally,the calculation of the broker's expected profits is given.(2)Then,in the environment without competition,we analyze how cloud broker allocates resource instances.We consider all arriving users' demands at each stage as a bundle,and propose a Q-DP algorithm to solve the resource allocation problem of cloud broker in the environment without competition.First,we model the process of the broker allocating resources to the bundle of users as a Markov Decision Process.We then use the reinforcement learning algorithm — — Q-learning algorithm to generate the allocation strategy for each state,which will decide how many resource instances will be allocated to the bundle of users.Next,we use dynamic programming to decide which cloud user's demands will be satisfied.Finally,we run experiment to evaluate the allocation strategy,and the result show that the resource allocation strategy used in Q-DP algorithm can provide more profits for cloud broker in the environment with limited resources.(3)Finally,we analyze how broker allocates resource instances in the competing environment with two cloud brokers.In this thesis,the process of competition between the two cloud brokers is modeled as Markov game.And then use the Minimax-Q learning algorithm is used to generate multiple resource allocation strategies(including the pricing strategies)of cloud brokers in the competing environment.And then we run experiment to evaluate the allocation strategy.The experimental results show that pricing strategies generated by Minimax-Q learning algorithm is helpful to improve the long-term profits of cloud brokers.In addition,allocation strategy generated by combining Minimax-Q learning algorithm and Q-learning algorithm can also improve the long-term profits of cloud brokers in a competitive environment.The results in this thesis can provide some useful insights for the cloud brokers to allocate limited resource when the users' demands are stochastic.