| Nowadays, we have entered into highly information-oriented society, and every important area involves data storage, data processing and data analysis, such as astrophysical research, bio-science and the Internet which we use every day. All of these behaviors will generate large data, which can easily exceed the traditional computer storage and analysis capabilities. Hadoop is an open source distributed computing platform for large data analysis, and it supports massive data storage and distributed parallel computing. Currently, the large-scale data-intensive computing platform, such as Hadoop, has been got widespread attention and applied by domestic and foreign IT companies.However, for such a large data management, counters often require professional training. In order to reduce the cost of data management, and make counters manage data easier and more convenient, Facebook developed a closed source tool HiPal, which is a graphical tool. HiPal is capable of direct dialogue with Hive, to complete data discovery, query editor, graphics and dashboard functionality to create, etc.But HiPal is only confined to the Hive operation, which is not able to deal with cross-platform data friendly and competent. This thesis proposes a graphical tool that can handle massive data, and make counters manage data easily. The system architecture is a CS model, with background data processed by Hadoop. Between Hadoop platform and client, co-ordination service server is designed for communicating client and the underlying Hadoop. Client is a graphical interface, by which users can manage data, design workflow and manage task. The server is consists of the data management module, workflow analysis module and schedule task management module for parsing and responding to client requests. System working mode is a client sends requests to the server, and for different request types of client sending through the server will be parsed and processed by different server module. |
PDF Full Text Request