Design And Implementation Of Edge Internet Of Things Data Stream Processing Platform And Task Scheduling System

As a newly emerging computing paradigm,edge computing aims to establish a relatively comprehensive application platform with core capabilities(such as computing and storage)close to the terminal devices of the Internet of Things(Io T).On this basis,the real-time Io T data reported by terminal devices can be fully utilized for rapid processing and analysis,so as to reduce the computing and storage costs of the Cloud and provide edge-side users with multiple intelligent application services.As Io T integrates with next-generation information technologies such as machine learning,big data computing,etc.,the Io T applications need to process a large number of real-time data flow.At the same time,challenges concerning the users’ delay requirements for real-time Io T data processing and the demand for application intelligence have been brought to conventional edge computing platforms.Due to the limited cluster resources of edge computing,a distributed parallel data processing workflow needs to be established to make full use of these resources.Moreover,tasks should be effectively scheduled and arranged,so as to improve the overall operating efficiency of edge applications.In this paper,a five-layer software architecture of distributed Edge Io T data processing platform was established using the container orchestration technology.And real-time data processing and machine learning applications on the edge were achieved in combination with the mainstream big data computing and artificial intelligence frameworks.In addition,an intelligent scheduling system and process was developed for online machine learning tasks in the platform.The main work of this thesis can be summarized as follows:1.A five-layer software architecture of data processing platform of Io T Edge based on con-tainer orchestration framework,Kubernetes,was proposed,and a comprehensive real-time Io T data processing workflow was designed.The software architecture was mainly divided into Io T data access layer,data transfer layer,big data processing layer,online machine learning application layer,data storage and visualization layer.2.The distributed edge computing software platform designed in this thesis was deployed on the cluster of heterogeneous edge servers.Big data real-time power computation and LSTM-model-based online power prediction applications were realized by applying software plat-form to the real-world smart-grid scenario.Some aspects such as processing delay of Io T data flow,cluster computing resource usage and scalability of the platform were tested.3.An online scheduling framework of edge machine learning tasks was designed and realized based on deep reinforcement learning(DRL).First,an offline task scheduling algorithm based on Deep Q Network(DQN)was designed,and the convergence of this algorithm was validated.Then,a set of scheduling control plane components based on Kubernetes con-troller mode were developed.Periodic dynamic scheduling control flow was realized by combining DQN reinforcement learning scheduling algorithm and Kafka message middle-ware.The scheduling framework was further verified to be capable of effectively reducing the average completion time of online machine learning tasks through the comparative ex-periments.Finally,a scheduling information management platform was designed and im-plemented.