Design And Implementation Of Scalable Metadata System For Cloud Storage Gateway

As the processing of big data applications is becoming the trends of cloud services,mass data storage platform is confronted with great challenges as the core infrastructureof all kinds of Internet applications. On one side, the scale of mass data storageplatfomrs and the amount of data they processed are increasing rapidly. On the otherside, the difference among data and applications grows more clear, which needs storagesystems of different types to optimize the efficiency of storage and access. As anintermediate infrastructure between cloud storage systems and local applications, cloudstorage gateway has great advantages in processing large scale different types of data.Nevertheness, most present cloud storage gateways are designed specifically for onetype of storage system and not open source, which makes it difficult for them toaccomplish their real concepts and goals.To solve the above problem, our group has addressed a generic distributed storageachitecture--JoinIn Cloud Storage Gateway, which supports WAN applications. JoinIncan afford users with convenient storage services with different types of storage systems.metadata and real data are maintained seperately in JoinIn. Metadata system maintainsthe important metadata of the system, which makes metadata systems very important tothe whole system. In ultra large scale storage systems, the design of metadata systems isconfronted with two serious issues: service performance and scalability. The hugeamount of accesses raise strict requirement. And the massive amount of metadata raisesstrict challenge.To solve the service performance problem, we designed a lightweght metadatasystem. A lightweight inode structure can reduce the load on metadata system, whichcan increase the access processing rate. We divide files into blocks and designed a fileincremental update method which can reduce network transmission overhead whilereading and updating files.To solve the scalability problem, a variety of factors that may influence loadbalance and locality have been analyzed. We addressd a metadata distributing policybased on dynamic subtree. In our metadata distributing policy, metadata can bedistributed evenly to every nodes and relevant metadata can be allocated to the samenode, which maintains metadata locality while achiving load balance.At last we implemented a prototype of JoinIn with Tencent cloud computingexperiment platform as remote storage system, and compared accessing remote storagesystem via JoinIn with accessing remote storage system via fuse-hdfs. JoinIn cansignificantly simplify the accessing interface of cloud storage systems. At the same time,JoinIn can increase read and write throughput by4.2%and3.7%respectively, anddecrease read and write latency by4%and3.6%respectively.