Design And Implementation Of Service Scheduling Management System Based On Hadoop Platform

With the development of digital information age,the global digital information resources are increasingly large.Big data penetrating into all walks of life,becomes a very important factor of production.Due to contrary relationship between efficient processing requirements and processing costs for big data,service providers of big data computing platform provide users with convenient and lower cost of computing service mode.Service providers establish a common big data platform based on the existing big data technology.At present,the mainstream big data platform is built based on Hadoop technology.For big data applications,Hadoop has become a very mature technology,which has a special performance especially in the efficient parallel computing.However,it still has shortcoming in the actual application of Hadoop.Firstly,for the common Hadoop platform of multi-service,the service provider cannot manage multiple services effectively.Secondly,in the existing service scheduling technology,there is no an efficient service scheduling strategy to achieve the maximum resource utilization of the platform and meet the needs of computing services at the greatest extent,which cannot make service providers get the maximum profit,but also lead to the resource utilization declines.What's more,in the process of services executed,an integration monitoring of platform resources and service status is not implemented.And the service execution process and resource consumption are not visualized.The platform and services are not monitored fully so that troubleshooting problems is more difficult and abnormal situations are not real-time warning.To sum up,it is an effective means that a management system of service scheduling optimization for big data applications is set up to solve the current problems.From a service provider of Hadoop platform point of view,a service scheduling management system for Hadoop platform is set up.The system is divided into four modules:service management and maintenance module,service scheduling optimization module,service timing execution module and service visualization monitoring module.When services are submitted to the system,the system manages service status,attributes,etc.according to service management requirements.Aiming at the existing defects of service schedulers,a service scheduler is proposed that can simultaneously satisfy resources,services,and revenue requirements based on reward and punishment coexistence revenue model.And a forecast model of future available resources is proposed based on a scheduling strategy generated by the service scheduler and current platform resource usage.The services are added the scheduling strategy that meet the remaining resources to maximize service provider revenues.The services are executed by manual timing or automatic timing based on service execution requirements and scheduling strategies to increase the flexibility of service execution.At last,the collected server resource data,platform resource data,and service execution status data are presented to the system interfaces in a visual manner.This system solves the main problems of the Hadoop platform in practical applications.It strengthens the management and maintenance of multi-service,schedules a number of off-line services reasonably to meet the requirement of quality of service at the greatest extent,achieve the goal of the maximization resources utilization of the platform and the revenue of service providers.Moreover,the resources of servers and the platform and the services execution are monitored visually to grasp resource changes timely and do real-time warning of abnormal situations.The system is designed and constructed according to the process of the service submission,service scheduling,service execution and resources and services visual monitoring.The realization of the whole system improves the management and execution efficiency of the service greatly and saves resources cost,which is very meaningful and valuable.