Batch Task Scheduling With Security Constraints In MapReduce

Task scheduling is an important factor influencing the performance of MapReduce frame-work for distributed computing.It has become a hotspot in recent years.The independent periodic execution of the batch task is an important task type in the MapReduce environment.When dealing with big data processing of public security data or financial data,security require-ments become important constraints that can not be ignored.This thesis studies the batch task scheduling method with security constraints in MapReduce environment,which has important theoretical significance and application prospect.In this thesis,we consider the problem of MapReduce task scheduling with security con-straints.Considering the Map and Reduce phases of security and data locality constraints,the object of optimization is to minimize the batch task completion time,taking into account job fair-ness and cluster load balancing.A two-stage task scheduling mathematical model with security constraints is proposed.Based on this model,a batch MapReduce task scheduling method with security constraints is proposed.The method consists of three parts:Map task matchmaking,Reduce task matchmaking and scheduling sequence adjustment.In the Map task scheduling phase,two kinds of Map task matchmaking operators with security constraints and load bal-ancing control are proposed to obtain the Map task scheduling sequence;In the Reduce task scheduling phase,two kinds of data locality aware Reduce task matchmaking operators with security constraints are proposed to obtain the sequence of Map and Reduce task scheduling;In order to optimize the scheduling sequence to achieve the purpose of further shortening the maxi-mum completion time of the job,four neighborhood structures based on insertion and exchange are designed.The scheduling sequence obtained by Map and Reduce matchmaking operator is used as the initial sequence,and then the local search adjustment method based on different neighborhood structures is proposed.To verify the efficiency and effectiveness of the proposed algorithm,we adopt analysis of variance technique to analyze parameters and components of the algorithm to get the best appropriate values of the considered problem.The proposed algorithm is compared with a fair scheduler on instances of different cluster and job scales.The experimental results show that the proposed method has a significant effects under different cluster sizes and job scales.The proposed algorithm is superior to the fair scheduling in terms of completion time,data security and load balancing.