Research And Application Of Small File Storage Technology Of Massive Animation Resources

With the development of massive data,technical bottleneck of animation production is mainly embedded in the follow aspects such as the limit of underlying computational resources,weak storage capacity,low transmission rate.Recent years,the emerging cloud computing architecture provides effective ways and means in order to solve the above problems.Establishing new "Anime Cloud" by combining cloud computing and anime domain will be the mainstream direction of future animation production technological development.Anime material library is a large-scale repository with scientific and practical.With the continuously improvement of the coverage of all kinds of professional material,efficiency of material provided platform and user experience requirement,establishing a cloud-based anime material library will be the inevitable growing trend.Hadoop Distributed File System(HDFS),a cloud storage platform,provides high-performance,high reliability and high expansible storage service to mess data.HDFS fits to store big streaming files.Huge small files increase HDFS's internal storage and influence its expansibility and performance,for lots of small files name node can occupy vase memory.This paper firstly analyzed existing solutions to HDFS storing mass small files,and deeply researched distributed environment,working principle and usages of Hadoop.And on this basis,widely used mass small files storing schemes based HDFS were improved.Small files are grading deal according to file size,the data of mass small files was equilibrium distributed by using linear hash indexing technology,and created linear hash index for the combined small files at the same level.Anime material database system index will append files stored in HDFS to data warehouse tools(Hive)through fixed time cycle,and put data structure via the form of table to improve HDFS's ability to handle mass small files and to effectively reduce the name nodes' load pressure.Putting the improved mass small files storing scheme into anime material database,the test and analysis of experiment data showed that our scheme can effectively reduce mass small files written time and make the data of mass small files evenly distributed by using grading linear hash index method.Users get data warehouse index results through Hadoop visualization analyze(Hue),which makes memory overhead and storage space effectively reduced.