Design And Implementation Of Task Scheduling In Distributed Computing Framework Antnest

With the rapid development of computer network technology in recent years, Internetcompanies provide users with more and more applications and personalized service,followed by mass data submited by users. Enterprises must process the data promptly andcorrectly so as to feedback results to users. Usually people would like to a distributedcomputing system to accomplish these complex tasks. However, traditional distributedcomputing platforms often not only need to be configured with large-scale server clusters,but also require professional technical staff to follow up and keep maintenance, that is abig cost for many companies.The lightweight distributed computing framework Antnest adopts client/servercomputing model and three-tier architecture. Antnest can deploy huge task in cheapdistributed task nodes. The center node is responsible for the scheduling of the entiresystem, task nodes execute task using the multi-JVM and multithreading. To address thisissue, we designed a multi-queue preemptive scheduling algorithm for Antnest, whichfully taken tasks’ requirements on real-time into account, high-priority task can seize thesystem, the preempted task will be moved to the high priority task queue waiting for thescheduling again, while utilizing the plan_start_time of the task to ensure the effectivenessof the task execution. What’s more, Antnest designed a dynamic task allocation algorithmto complete the allocation of subtasks. In the dynamic allocation algorithm, thedistribution relationship between the sub-tasks and task nodes is determined dynamicallyduring the execution of the task, Antnest would give priority to assigning subtask to thetask node who had finished its subtasks earliest. In the implementation of Antnest taskscheduling system, we use user-defined communication protocol to complete the messagecommunication with each other, and trigger the task logic with a specified task interface.Task node state management mechanism would complete the resource management andensure system’s scalability. Meanwhile, the heartbeat detection and subtask retransmissionfault-tolerant mechanism could ensure the correctness of task execution.The test results show that Antnest system can improve the implementation efficiencyof the task and shorten the execution time effectively. At the same time, the results alsoverify the rationality and feasibility of the multi-queue preemptive scheduling algorithmused in Antnest.