Research And Implementation Of Task Scheduling Framework In Distributed Environment

In today’s era of big data,in the face of increasing computing demands,distributed computing has become a research hotspot in various fields.A large number of computing requirements often contain huge computing logic.When many small computing tasks are obtained by decomposing such computing requirements,the distributed task scheduling framework can be used to allocate these computing tasks to each computing node to achieve task concurrency processing.Model this huge calculation logic into a workflow,and the numerous calculation tasks in the calculation logic are regarded as interlocking stages in the workflow.Therefore,it is particularly important to design a distributed task scheduling framework to deal with this kind of workflow.Based on the distributed environment and the load of each working node,a distributed task scheduling framework(DTSF)is proposed.The framework uses Hadoop Distributed File System(HDFS)as a distributed file storage system.The communication of each node adopts the Client/Server(C/S)architecture.The working nodes use virtualization technology to package the processing programs needed in the working nodes.The master node receives the workflow request submitted by the client,analyzes the workflow,and dynamically allocates the parsed tasks according to the load of each worker node,and the framework can deal with soft failures in the execution of the workflow to a certain extent.Experimental results show that DTSF can correctly complete the entire process of workflow submission,scheduling,and execution.When the working node or the master node fails,the workflow managed by the node will be completed normally when the node is working again,indicating that the robustness of the framework is good.