Scheduling For Fast Big Data Analysis In Geo-Distributed Cloud

With cloud service rapidly developing and widely using in the academic world and industrial circles,the geo-distributed style is evolved as the main fashion in the cloud system.Cloud service providers are required to ensure the QoS requirement,which can be an another challenge in nowadays ages.In geo-distributed cloud,the bottleneck of uneven computing resources and limited network resources is a common view that causes performance improvement lags in the cloud system.The options,which are excluded if they will lead to extra costs,can be utilized to improve execution efficiency.Through the consideration of factors mentioned above,we propose the Community Detection-based Scheduling(CDS)algorithm as the scheduling strategy and the File-TransferScheduling(FTS)algorithm as the bandwidth allocating strategy.The CDS algorithm schedules tasks based on the dependency relations between task,data and data center.The FTS algorithm allocates bandwidth based on the fact of task computing time and data transmission amount.The CDS and FTS algorithm treats the problem of minimizing the global completion time as the ultimate goal.The real China-Astronomy-Cloud network and the Google's data center network are used to evaluate the performance of CDS and FTS algorithm from the perspective of minimizing the total file transfer volume,minimizing the global completion time,the extreme case,the distribution of global completion time and the bandwidth usage.According to experimental data,we reduce the total file transfer volume by up to 40.7%,and the global completion time by up to 35.8%.