Spark Parallel Dynamic Programming And Its Application In Optimal Generation Dispatching Of Multi-Reservoir System

Development of hydropower energy provides a solid foundation for the sustainable development of modern society, but the utilization of hydroelectric energy is low and there is still much room for improvement. Rearch on reservoir(s) generation optimization is an important project to improve the energy efficiency of hydropower, the primary research methods are establishing new models and improving algorithm for solving.With the expansion of hydropower station’s size, traditional algorithm will be limited by the problem of "curse of dimensionality". A newly research on increasing computing power by applying parallel computing techniques has been promoted recently. Existing parallel reservoir scheduling algorithms are mainly based on framework of Fork/Join or MPI: Fork/Join is designed for single machine which is the limitation for large-scale problems; MPI can be constructed on cluster, but programming for MPI is a tedious work that many issues need to be considered besides the algorithm itself and MPI is weak on fault-tollerent so it can’t ensure the reliability of the calculation.In recent years, a variety of cluster computing engine have provided solutions for massive data processing, among them Spark provides better support for iterative jobs on distributed datasets and is widely used for machine learning, data mining and image processing with massive data. Spark encapsulate data segmentation, parallel processing, fault tolerance and so on, these makes programmer easier to program. Spark uses information about how datasets was derived from other datasets(Lineage) to recover lost data which provides higher data reliability. In this paper, several parallel processing of dynamic programming and discrete differential dynamic programing algorithm has been made:(1) After studying the state transition of dynamic programming and the operating mechanism of Spark, an iterative calculation of Key-Value data sets is used to present the process of multi-stage transfer. Spark parallel DP and DDDP were implemented and have been experimented in multi-core computer environment.(2) When Spark parallel DP runs on heterogeneous Spark cluster, it comes to a problem that tasks in the same stage can’t be finished at the same time, thus a task allocation method based on node computing capacity assessment was proposed and has been tested in a heterogeneous cluster environment.The results show that Spark parallel DP and DDDP can effectively improve the efficiency of the algorithm; Spark can break the speedup bottleneck of Fork/Join on parallel DDDP in cluster environment; the new task allocation method can solve the problem where completion of tasks in the same stage are out of sync and improve computing efficiency. Considering saving time and reducing costs, Spark has a broader application prospect than other parallel computing frameworks. The basis of theory and method was provided for the further studies of cloud computing in optimal reservoir system operation.