Energy-Saving Control Method For HVAC Based On Reinforcement Learning

Worldwide,40% of society’s total energy consumption comes from the building industry.In China,the energy consumption of the building industry is close to 20.6% of the total social energy consumption.At the same time,more than 50% of the energy consumption in the building industry comes from Heating Ventilation and Air Conditioning(HVAC).Therefore,it is very important to develop and implement effective energy saving control technology for the society.At present,HVAC energy saving control methods are mainly divided into model-based control methods and model-free control methods.Model-based control methods have been widely studied and verified by scholars,but they are highly dependent on accurate simulation models.In contrast,model-free control method does not need a lot of historical data to build an accurate simulation model.Model-free control methods include rule-based control methods and reinforcement learning-based control methods.Rule-based control methods are usually static,and the development of control rules depends on the experience of engineers or device administrators,while reinforcement learning control methods do not depend on the experience of engineers and can carry out adaptive learning.Therefore,for the old buildings lacking historical data,the control method based on reinforcement learning has more research value.This paper focuses on how to use reinforcement learning algorithm to study the HVAC optimal control method which lacks the equipment operation history data and equipment performance model.Based on the energy consumption analysis of the actual case system,this paper constructed the HVAC mathematical simulation platform,and used the reinforcement learning algorithm to optimize the combined control of cooling load distribution mode,cooling water pump frequency and cooling tower fan frequency of different types of chillers in the case system.The main content includes the following three parts:(1)Aiming at the lack of equipment running history data and equipment performance model in the actual case system,a mathematical simulation platform is constructed.The lack of historical data in the case system makes the equipment performance model hard to be regressed in a data-driven manner.The mathematical simulation platform builds mathematical models of equipment such as chillers,cooling towers,and cooling water pumps by integrating the information in the equipment nameplate of the case system into the standard equipment models in Energy Plus,and according to the layout information of the case system,each equipment combination is designed.The coupling relationship between cooling water outlet temperature and cooling water return temperature is treated by iterative circulation.The results show that the mathematical simulation platform is reasonably designed and can be used as a platform for pre-training and experimentation of reinforcement learning algorithms.(2)For the problem that the performance of the model-based control method is highly dependent on the accuracy of the simulation platform,a single-agent control method based on DQN is proposed.The method is pre-trained on a mathematical simulation platform and then deployed into the actual system for adaptive learning without requiring the simulation platform to have a high degree of accuracy.In actual deployment,due to the uneven distribution of training samples and the formation of a large-scale action space by combining multiple control points,the strategy learning of the single-agent control method based on DQN is slow.Aiming at this problem,this paper proposes a single-agent control method based on double-pools DQN based on the single-agent control method based on DQN.In this method,two experience pools are designed to store different types of training samples and select samples for training according to a specific proportion.In addition,the algorithm optimizes behavioral policies and compresses the choice space of actions through taxonomy.The simulation results show that the control algorithm based on the double-pools DQN has a significant improvement in the policy learning speed.(3)In order to further improve the speed of learning strategy based on single-agent reinforcement learning algorithm,a deep reinforcement learning method based on cooperative multi-agent is proposed.The method establishes multiple sub-agents to optimize each node respectively,and compresses the action space formed by the arrangement and combination of single-agent deep reinforcement learning algorithm into a joint action space of linear combination of multiple agents,which greatly reduces the action space.Rewards are shared between agents to reduce the total energy consumption of the system.However,the cooperative multi-agent is affected by environmental instability,resulting in inaccurate estimation of the early value function.Aiming at this problem,a pipeline communication mechanism is proposed based on the cooperative multi-agent deep reinforcement learning algorithm.The mechanism prioritizes all agents and transmits decision information between agents through pipelines to reduce the impact of environmental instability.The simulation results show that in the early stage of algorithm training,the multi-agent deep reinforcement learning algorithm based on pipe works better.