Design And Implementation Of Big Data Platform For Data Aggregation And Storage

Data have always been one of the most important assets for enterprises.Many enterprises have spent a lot of human,material and financial resources to build a variety of information systems to manage their data assets.However,the heterogeneity and dispersion of the original data assets make it very difficult for data aggregation and analysis.Therefore,the aggregation and persistent storage of data in heterogeneous information systems has become an inevitable choice for enterprise to break the silos of data,and is the basis for enterprises to create value through data.In this context,the big data platform for data aggregation and storage is designed and implemented.The data migration task management method based on the unified data migration framework is proposed in this thesis,which could uniformly manage data migration tasks from relational databases or other big data platforms to this big data platform;Then,the data storage capacity prediction method based on the time series trend analysis,which avoids problems such as downtime of the big data platform and interruption of business applications due to the exhaustion of storage capacity,is brought forward;Next,the data migration task scheduling method based on the ATSP-TP algorithm is specified,which could be used to optimize the operating efficiency of the data migration tasks on the big data platform.In the ATSP-TP algorithm,the available capacity of the platform in the future is obtained by the prediction of the data storage capacity,the free time period of the platform in the future is obtained by the prediction of the available time-slot of the platform,the expected time and space requirements of a new data migration task is obtained by the prediction of the data migration task time-slot demand and the optimized scheduling for a new data migration task is realized by the data migration task placement using the data of the above three.The effectiveness of the platform is verified by a set of functional and non-functional tests on the platform.