Some Research On On-Line Index For Dynamic Text

Inverted index is an important technique to improve search efficiency, how to balance the space efficiency, dynamic performance, and retrieve efficiency are the main problems that inverted index faces. In this paper, we concentrate on index building, updating, compression storage and search retrieve under dynamic environment to improve the overall performance of retrieval system.The paper first introduces the research background and significance of the topic, analyzes the structure model, compression, and update maintain of online index at home and abroad, and proposes the main content of this research.Next we put emphasis on analyzing the storage problems of online index, design the four-level structure, and achieve the file system of the online index. File system includes barrels file(barrels), storage field file(fdd), storage field link file(fdl), dictionary table file(voc), and term frequency file(fot).Then focus on the problems like lack of mass data storage, and inconvenience of query traditional compress file, add, modify and delete element. After analyze Golomb code and BIC code, propose bit-oriented operation dynamic block self-index compression(Bit_DIB) and byte-oriented operation dynamic block self-index compression(Byte_DIB).Subsequently, this paper makes the research problem of update maintain of online index, we design index update algorithm based on DIB self-index compression structure, the algorithm suitable to small amount of the text content update situation. Then focus on index merger problem, propose four principles to be followed by re-merge, analyze four index maintain methods:re-build, re-merge, in-place and hybrid approach. After conclude GP, LOG, DBT merger strategy, we design based on N-level class state diagram(NSD)index merge algorithm.Finally, on the basis of theory research, we design and implement online index tools prototype system, provide experimental platform for follow-up study.