Design And Implementation Of The Loading Technology Of Massive Text Data

With the increasing development of internet, using of internet data are noticed more and more by people. How to store and manage the massive text information has become a inevitable problem of the data mining, search engine, network management, network security and network information collection. The massive text date produced form network has some sharp-cut characteristics: full-text retrieval needed, high-speed, high density, continuous, great scale. It turns into a task to store and manage this type of massive text data.On the base of analyzing the characteristics of this massive text data, the thesis using the distributed object middleware, ORACLE 10G exchange partition feature and parallel-job scheduling arithmetic, and then it designs and implements a loading system of massive text data developed for StarTPMonitor.First of all, with the compare and analysis of every solution project, this paper give the architecture of the loading system of massive text data based on parallel-job scheduling, afterward it describes that how to solute the problems such as the high-performance, high availability of loading of massive text data. Then, this thesis implements this system by the design of the architecture, and chiefly describes the parallel-job scheduling model of the loading system, implement the coordinated scheduling between the loading tasks in multi-resource database system.At last, the practical tests indicate that the function characteristics and the performance of this loading system are better than the target we desired. Beyond the proper scheduling degree, this loading system of the massive text data based on parallel-job scheduling arithmetic is much better than the classic loading system.