Model-free Optimal Control Method Based On Supervisor Group And Multi-objective Optimization For Central Chiller Plants

Building heating,ventilation and air-conditioning(HVAC)systems consume over60% energy during building operation.In order to conserve building energy consumption and reduce carbon emission,the operation of building HVAC systems,especially central chiller plants need to be optimized.Currently in engineering practice,central chiller plants are typically controlled by simple logics:(1)system on-off status is manually controlled by on-site operators;(2)in some cases,HVAC appliances including pumps,chillers,cooling towers operate at nominal frequencies or nominal set points;(3)sometimes frequencies of the appliances above are controlled by local feedback controllers with simple PID logic.The performance of control logics above highly relies on engineers’ commissioning experience.Hence,energy conservation potential in HVAC engineering practices could be further utilized by optimizing system operation.In order to realize that,a lot of researchers carried out studies on model-based optimal control methods for HVAC systems.With accurate system performance models and efficient optimization algorithms,such methods could achieve great performance on system energy saving.However,the performance of this kind of methods highly relies on the accurate system model,which takes a priori knowledge,sensor implementation integrity,historical system operation data,and manual labor to establish.The high requirements of model-based optimal control limit its application in engineering practices.As for the framework of control systems,the widely studied centralized control scheme relies on complicated system model and difficult optimization process,while complete decentralized control scheme leads to complex interaction logic among computational nodes.Motivated by the problems above,a model-free optimal control method based on supervisor group(i.e.,multiple agents)and multi-objective optimization for electric central chiller plants is proposed and investigated in this study,to(1)enhance the feasibility of optimal control methods in engineering practices;(2)cut down the application cost of the optimal control technique including manual labor and sensor implementation;(3)reduce the dependency of optimal control on system models;(4)evade the control risk caused by model uncertainties;(5)realize customized optimal control for different types of buildings.This study is composed of the following parts:(1)An agent-based,modular control framework is proposed for central chiller plants.In this framework,a central chiller plant is divided into several equipment groups(chiller group,chilled water pump group,etc.);each equipment group is equipped with one virtual optimal controller(agent),and each agent is in charge in its own autonomous region(the control task of the conventional central optimal controller is disassembled to these agents).This agent-based control framework could reduce the computation complexity of the conventional central optimization;besides the complex interaction mechanism of extreme decentralized control(i.e.,each single equipment is regarded as one agent)could be evaded.(2)Based on reinforcement learning(RL)technique and utility functions,a series of model-free optimal control methods are proposed for control agents of each equipment group.Each agent could independently work on itself.These proposed methods could relieve the reliance of controllers on system performance models,because with the proposed methods,control agents continuously learn how to better control equipment groups during their interaction with the environment.Moreover,on the basis of classical RL algorithms and HVAC systems’ operation feature,a linear time policy and expert knowledge interpretation are proposed to accelerate agents’ learning process,and to enhance the control stability and robustness.In order to verify the control performance of the proposed method,a hybrid system model,composed of black-box and white-box equipment models,is established as a virtual environment for controller test.The hybrid system model is set up and calibrated using the actual operational data of a real HVAC system.Comparative simulation case studies are carried out by implementing different controllers including basic constant speed control,local feedback control,model-based optimal control and the proposed model-free control method to control the operation of the virtual system.Simulation results indicate that the proposed model-free method could save more energy than basic constant speed control and typical local feedback control.Moreover,the control performance of the proposed method could continuously evolve during the interaction between agents and the environment.In the second cooling season,the performance of the model-free controller would be close to the model-based controller.(3)During the operation of central chiller plants,different types of appliances work coordinately.When optimizing different equipment groups,the influence among each group needs to be carefully handled.In order to realize the optimized operation of the whole central chiller plant,it is necessary to optimize multiple equipment groups simultaneously.Meanwhile,since the model-free control methods proposed in this study are based on RL algorithms,each agent has to learn and evolve from the environment feedback;in this situation,the simultaneous function of multiple agents would lead to dynamic environment and mutual inference.Points above reflect a problem in the model-free control: how to harmonize agents’ objectives and realize coordinated model-free optimal control to central chiller plants under potential mutual inference among each control agent?In order to tackle the above problem,a quantitative analysis is conducted on the established virtual environment to investigate the mutual inference between cooling tower agent and cooling water pump agent.Further,with the knowledge of multi-agent reinforcement learning(MARL)and game theory,this coordinated model-free optimization problem is described and categorized into two types: fully cooperative problem and non-fully cooperative problem.For the first kind of problem,different MARL mechanisms(multiplication,division,and interaction)are respectively realized with the proposed model-free controller to control the established virtual environment.Simulation results show that interaction and multiplication mechanisms could make the model-free controller reach higher upper limit of control performance than the division mechanism.And interaction and multiplication mechanisms take more time for control agents to learn and evolve.While the division mechanism is on the contrary.For the second problem,a model-free coordinated control method is designed by combining an equilibrium solving method and Sarsa RL algorithm,to coordinate all agents’ objectives.The proposed method is compared with a mature MARL algorithm(Win or Learn Fast-Policy Hill Climbing,Wo LF-PHC)and typical local feedback controller,in the virtual environment.Simulation results suggest that the proposed method performs better than typical local feedback controller.And the proposed method could reach control performance close to Wo LF-PHC,while with simpler calculation process,less undetermined parameters and less potential risk from parameter tuning.Hence,the proposed method is more user-friendly than Wo LF-PHC.(4)Finally,the model-free chiller control method proposed in this thesis is applied to two real HVAC systems: one is in a factory in Jiangsu province,the other is in an office building in Shanghai city.In the factory system,the model-free method is compared with an SMP(simplified multivariate polynomial)model-based chiller control method;in the office building,the proposed model-free method is compared with artificial expert manual control.The application results indicate that after one month’s learning,the performance of the proposed model-free method could reach acceptable performance(energy conservation and indoor comfort maintaining)close to SMP model-based controller and artificial expert manual control.Moreover,the proposed method requires less pre-conditions such as manual supervision and model establishment,which emphasizes its feasibility in engineering application.To tackle the optimal control problem in central chiller plants,an agent-based modular control framework is proposed in this study,along with a series of model-free optimal control methods for each single control agent.Moreover,in order to realize multi-agent model-free control in central chiller plants,a model-free coordinated control method is proposed based on RL technique and game theory.Finally,the proposed model-free control method is applied to two real HVAC systems and simulation virtual systems to validate its control performance.The simulation case study and engineering application prove that(1)with the online learning mechanism of RL,the proposed method could reduce the dependency of optimal control on system models and enhance the feasibility of optimal control in engineering applications;(2)the proposed expert knowledge interpretation and time linear policy could accelerate RL agent’s learning process and control stability;(3)with less application cost and requirements,the proposed method could reach control performance close to modelbased control method and artificial expert manual control,better than typical local feedback control.