Design And Implementation Of Massive File System With High Scalability

With the BIG DATA approaching, the scale of data has beenexploding and meanwhile applications of data takes on an air ofprosperity, which naturally brings the storage system’s development withnew opportunities and challenges. The obsolete method—scale-up, can’tcope with the new applications’ requirements: huge volume, highconcurrency and low costs; rather, the scale-out method which aggregatescommodity computers with competent distributed scheduling policiesworks well.BlueOcean file system is a scale-out massive distributed file system,which performs well as a storage stack for the batch processing systemsuch as Hadoop. However, there is a hard nut to crack when BlueOceanacts as a traditional file system. Shortage in scalability, availability andperformance is impeding the BlueOcean’s development remarkably, lowscalability in particular. This project has an ambition to partition theunited meta data into sub-trees, which can eradicate the root of all evildoomed with the notorious problem—single point of metadata node. All the endeavor is mentioned below: splitting the dentries intosub-trees arms the system with the capability of distributing metadata ondifferent nodes; designing the monitor nodes makes tasks of arbitratingmetadata nodes’ status, collecting and handing out information onsub-trees, and changing the scale of the metadata node clusterdynamically feasible; optimizing the model with which multiple datanodes interact with multiple metadata nodes, avoids the gigantic filedata’s movement resulting from the rename operation; regulating thedistributed transaction processing between separate nodes, contributes tohighly reliable migrations of metadata information.To conclude, the results of corresponding experiments illustrate thereis a great gain in the scalability of the BlueOcean file system through theendeavors mentioned above, what’s equally important the systemperforms well with the monitoring protocol and distributed transaction.