Data-driven Task Allocation For Multi-task Transfer Learning On The Edge

On edge devices,data scarcity occurs as a common problem due to the high cost of monitoring deployment and low reliability of measurement.Recently,transfer learning shows its effectiveness to tackle the data scarcity issue and serves as a widely-suggested remedy.It reuses parameters or training samples of source task to support target tasks which are lack of training data.However,the current multi-task transfer learning(MTL)systems are too computationally complicated for edge devices.The reason is twofold: 1)Machine learning(ML)model is computationally and communication-intensive;2)To avoid ML model being out-of-date and leverage the latest accumulated data,each task needs to be learned individually from scratch.Confronted with the computational challenge,it is urgent to solve how to improve the efficiency of MTL on the edge.To solve the above problem,the task importance,i.e.,the difference in the final decision performance(e.g.,energy saving)between the measuring task is executed and not executed,is first defined by analyzing the real-world MTL dataset.It is found that the importance of tasks obeys a long-tail distribution and fluctuates significantly in different environments.Based on this discovery,a cluster reinforcement learning(CRL)model is designed,which finds a similar environment by clustering algorithms from the dynamic environment set,and allocates tasks on this basis.This requires many environment observations to cover all possible situations.However,due to data scarcity on edge devices,the CRL model can confront with quite a few unseen environments.To this end,a data-driven collaborative task allocation(DCTA)mechanism is further designed.The core idea is to leverage the support vector machine model to predict the task importance,and dynamically adjust the task allocation decision of the CRL model based on real-time data.Experimental results based on real-world MTL dataset show that the proposed DCTA mechanism can reduce the processing time by more than 69% compared with the state-ofthe-art traditional MTL mechanisms.Besides,by applying this mechanism to an edge AIOps system,the effectiveness of the proposed DCTA mechanism is further proved.