Optimization Of Automated Container Terminal Yard Based On Reinforcement Learning

In recent years,with the continuous development automated equipment and artificial intelligence algorithms,automated container terminals have become the development direction of container terminals and occupied an important position in the intelligent process of the container terminal.Among them,the automated container yard not only has the characteristics of the traditional container yard such as huge amount of tasks,large variety and quantity of operating resources,operation difficulties caused by threedimensional placement,and the strong task dynamics,but also has timecritical requirements caused by automated equipment and interaction at both ends and crane cooperation caused by features perpendicular to the shore.Therefore,the container yard has become the bottleneck for the automated container terminal,and it is urgent to explore more reasonable operation plans and scheduling strategies.However,traditional methods such as exact solutions,heuristic rules,and intelligent algorithms have problems of long calculating time and insufficient solution,and are difficult to be applied in actual scenarios.So this article takes the automated container yard as the research object,and studies the corresponding operation plan and scheduling strategy based on novel reinforcement learning methods,which is of great significance to the intelligent process of the container terminal.The research in this article comes from Yangshan Phase IV automated container terminal yard.The main research contents are as follows:(1)A model for the optimization of the automated container yard is established.This paper analyzes in detail the four major characteristics of the container yard optimization problem: large-scale,a large number of complex characteristics and constraints,complex time-space coupling characteristics,and strong dynamics.Then the problem is decomposed into two progressive sub-problems: container space assignment and yard crane scheduling.Finally,problem assumptions,optimization goals and various constraints are established,and a mathematical model for the optimization of automated container yard is constructed.(2)A hyper-heuristic algorithm based on reinforcement learning for container space assignment is proposed.The algorithm includes two levels of content: low-level heuristics and high-level decision-making algorithm,and obtains better solutions through continuous improvement of the solution.In the low-level heuristics,local search operators,global search operators and intelligent algorithm operators with different characteristics are introduced.In the high-level decision-making algorithm,a strategy-based reinforcement learning algorithm is proposed to intelligently select appropriate low-level heuristics,and finally achieve the improvement of the performance.(3)An optimization algorithm of yard crane scheduling problem based on deep reinforcement learning is proposed.The algorithm is based on a sequence-to-sequence framework and uses a reasonable agent definition and interaction with the environment to find a better solution.Among the definition of the agent,the state,actions and rewards are defined based on different types of operations of the task,task feasibility based on crane conflicts and vehicle arrival time,and evaluation indicators.In the environmental interactive,the environment and state update methods are developed based on simulation and heuristic rules,and the strategy gradient calculation method and training method based on the attention mechanism are designed to achieve the performance improvement in the limited calculating time.At the same time,the model and algorithm proposed in this paper are verified through actual historical data and instances designed according to its trend.Compared with traditional methods,improved algorithms proposed in the literature,and actual heuristic algorithms,the algorithm based on reinforcement learning proposed in this paper can effectively improve work efficiency and reduce the vehicle waiting time.Finally,with the Yangshan Phase IV automated container yard as the background,this paper developed an automated container yard simulation verification system.Based on the actual needs obtained from the survey,the overall system design,functional module design and process design have been completed,providing effective tools and platforms to meet the requirements of automated container yards to improve overall work efficiency.