Data management, storage and access optimizations in high performance distributed environment

Effective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of distributed secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance or they sacrifice significant performance in exchange for ease-of-use and portability.; This dissertation discusses and presents a novel application development architecture which includes user applications, a Meta-data Management System (MDMS) and Storage Systems (SS) that provides a platform for both easy-to-use and automatic storage access optimizations. This dissertation discusses the that provides I/O optimization automation and a user API that provides users with a UNIX-like ease-of-use interface. MS-I/O, a Multi-Storage multi-storage resource architecture that can satisfy both performance and storage capacity requirements and employs a number of state-of-the-art I/O optimization schemes. As computing locally is crucial for high performance, a Distributed Parallel File System (DPFS) is designed to collect as many local storage resources as possible and create a parallel file system for high performance data access. To better evaluate the performance of accessing various storage systems, an I/O performance prediction mechanism is proposed and discussed. Finally, to address the problem of working environments which include a number of distributed computing and storage resources, an Integrated Java Graphical User Interface is designed to help users work efficiently and effectively.