An Active Storage System With Dynamic Task Assignment Policy

With the high-speed development of internet,the amount of data grows at a exponential rate and global information sharing become a reality.However,this exponential growth of data brought great pressure on information storage,transmission,and processing,and the applications of storage technology are faced with serious challenges.Active storage provoids a promising solution to addressing the limited I/O bandwidth issue for bigdata applications with growing data demands.The essential idea of active storage is to move computation close to data and reduce bandwidth requirement by offloading appropriate data-intensive operations to storage nodes,thus reduces the amount of data transferred and energy consumption.Prior research has investigated and deployed the concept in different forms but lack of flexible programming model and high adaptive task assignment policy.In this paper,we propose a novel Active storage architecture,AHDFS based on HDFS,it realizes fundamental functions of active storage concept and provides a flexible programming model.For ease-of-use of AHDFS,we implement the class ASInputStream which is extended from InputStream in JDK.The ASInputStream shields the details of the proposed system and provides a data access interface similar to the Java I/O.AHDFS provides a stream-based programming model for application program to design computation class to its requirements.In order to meet the varied scheduling demand of applications,AHDFS provides programming interfaces to design and submit the task assignment policy.Considering balancing the computation load between compute and storage nodes in active storage system according to the current system status is very important to make full use of idle computing resources of system and has been largely unexplored in prior research,we propose a dynamic task assignment policy,DTAP.It assigns computing tasks between compute and storage nodes according the feedback information of system environment including computing load of each nodes,the bandwidth of inner connection and the characteristics of offload computation.Our experimental results show that there is an optimal allocation proportion with the considerable impact of system status and characteristics of computation,and the proposed dynamic task assignment policy can make an allocation of computing tasks at a proportion close to the optimal assignment.