Studyon Grid File System With P2P Concepts

Data in a grid can be of any format and stored in any type of storage systems.There can be many hundreds of petabytes of data in grids, among which a very largepercentage is stored in ?les. A standard mechanism to describe and organize ?le-baseddata is essential for facilitating access to this large amount of data. We do research onresource location, dynamic replication and replica consistency and propose the archi-tecture of our Grid File System named TH-GridFS. TH-GridFS is a web-based virtual?le system with P2P concepts and with interfaces of a standard ?le system. There are two key pieces to TH-GridFS: the lookup mechanism for locating adesired ?le and the actual ?le download. The completely decentralized storage in P2Pconcepts merged into TH-GridFS and the stripped ?le transfer model based on dynamicreplication make the ?le transfer process inherently scalable. The hard part is ?ndingthe peer(s) from which to retrieve the desired ?les. Broadcast solution can not be scal-able for millions of peers, but DHT solution can. So we propose a new P2P overlaynetwork called GFS-Net to make lookup solution in TH-GridFS be scalable, ?nd ?lesquickly( after O(logn) hops, n denotes the network size) at the upmost and be resilientto the frequent arrival or departure of the participant peer(s). Files( including replicas) in TH-GridFS are not only readable but also writable.We introduce the Push Version-Number Consistency (PVNC) model and a writer/readerprotocol to implement it. PVNC eliminates nearly all read latency. A prototype of TH-GridFS has been implemented and applied in Chinese Edu-cational Resource Grid. It will help in facilitating access to educational resource ?lesdistributed all over China.