Design And Implementation Of Hierarchical Storage Function Based On Hadoop Distributed File System

With the coming of informantion age, total data amount is growing very big. To store t his amount fo data, traditional standalone storage system is now replaced by cluster storage system, The cluster scale is also growing very fast.But dasta has it own lifecycle. Data acc ess frequency will gradualy decrease as time goes by. Data stored more than three months b asically will not accessed again. While these rarely accessed data occupied 80% of the stora ge space.It will bring high level cost in storage system to store data in expensive and high-p erformance hard disks.HDFS is a widely used open-source cluster file system nowadays. this paper mainly focus on implementing data auto-leveling fuction at Name Node in HDFS.This function is implemented by three parts: data elimination algorithm, data migration, cold data storage management. Aging algorithm is used as the elimination algorithm in this paper. Also improved aging algorithm,the improved algorithm can reflect the factor of file access frequency in record cycle and file size, making aging algorithm well-applied in HDFS. By using data copy and delete operation in the original heart-beat communication mechanism of HDFS, data migration task is built and data migration is accomplished. The allocation strategy of DataNode is modified. Part of DataNodes uses optical library as storage medium to store cold data. These cold data DataNode is also managed in this function.By applying aging algorithm in HDFS, data tiered storage is accomplished. After some experiment, it is proved that data with lower access frequency in HDFS is automatically migrated to optical library DataNode.