Research On Dynamic Job Shop Scheduling Problem Based On Reinforcement Learning

With the deepening of economic globalization and the changing needs of customers,the external environment faced by manufacturing enterprises is becoming more and more complex and changeable.The classical static scheduling algorithm based on improving the optimization ability is difficult to function effectively in the actual manufacturing environment.In order to solve the scheduling problem in an uncertain and complex dynamic environment,a dynamic job-shop scheduling algorithm based on the combination of reinforcement learning and classical dispatching rules is studied in this thesis.The dynamic job scheduling problem in which jobs can arrive randomly is considered to establish effective solutions to job-shop scheduling problems with good optimization and self-adaptive ability in the unforeseen and dynamic environment.By analyzing the characteristics of dynamic scheduling problems and existing scheduling algorithms,a problem-solving framework that embeds classical dispatching rules into reinforcement learning algorithms is established in this thesis,and dynamic interference factors to characterize the dynamic events is introduced,such as random arrival of workpieces in the actual working environment.Meanwhile,the dynamic job scheduling problem model is established,which use the minimizing the maximum delay time as the optimization performance index of the scheduling algorithm to fit the needs of job production efficiency and customer order delivery demands.Aiming at the deficiency of the ability of classical rule scheduling algorithms to solve dynamic scheduling problems,the Q-learning algorithm in reinforcement learning is selected in this thesis,and an improved rule scheduling algorithm based on Q-Learning is also proposed.To improve the ability of the scheduling algorithm to explore and use rules,and obtain the best dispatching rule under each different job state,a new state-space representation method "Tard",a simple and effective reward mechanism,as well as a search strategy based on Boltzmann sampling function are established,where the action set is composed of classical rules.In order to verify the feasibility and effectiveness of the proposed algorithm,eight groups of benchmark scheduling examples are selected for simulation under the conditions of unexpected arrival of workpieces and five different urgency factors.The results show that the proposed algorithm could effectively shortens the maximum delay time.It also has good convergence,fast optimization ability and adaptability to the dynamic environment.However,the algorithm still has some problems,such as the short description of the real-time workshop state and poor stability in solving larger-scale scheduling problems,due to the constraints of Q-table dimension.In order to overcome the problems of the above algorithm,the deep reinforcement learning algorithm is further applied in this thesis to improve the rule scheduling algorithm,and a DDQN(Double-DQN)dynamic scheduling algorithm is proposed.By analyzing the main states of workpieces and machines in the production environment,eight scheduling state characteristics are designed.Meanwhile,three composite rules and seven single rules are selected to expand the original action rule base,add characteristic indexes related to delay time,and further enrich the reward mechanism.Besides,simulation has been done where 18 groups of classical scheduling examples are selected,in combination with three-time constraints,so as to verify the performance of the proposed algorithm in dealing with different scheduling problems.The simulation results show that the improved rule scheduling algorithm based on DDQN has further improved its performance on accuracy,robustness and generalization of scheduling solutions,and it could effectively solve job scheduling problems of different scales.