Realization Of A High Performance Data Retrieval System For Big Data Based On Lock Free Structure

The development trend of the processors has been developed from the high-frequency CPU to multi-core CPU,the average CPU number of desktop PC is expected be extended to 24 cores(or 16 cores32 threads)in early 2018,the mainstream server's CPU core number may also reach more than 12 cores.The large-capacity data retrieval system based on lock synchronization can't exploit the advantage of multi-core CPU in many aspects,and software development with lock synchronization is often prone to introduce serious defects,which increases the risk of causing serious problems such as deadlock and data competition.At the same time,the increase of the core CPU num leads to serious decline of the software's performance,accompanied with low scalability.In recent years,software developers have begun to research Lock-Free data structures.CAS-based fine-grained synchronization primitives avoid Lock-Free data structures deadlock and race condition.More importantly,it is extansible with the increase of core number.The industry also has provided a number of non-lock based data structure library.Applying such a structure to the retrieval system will significantly improve the performance of the retrieval.Based on the improvement of the existing electronic business retrieval system,this paper studies the realization of high-performance data retrieval system based on Lock-Free data structure technology.The paper will be summarized as the following:Firstly,the present developments and trend of high performance retrieval system are introduced in depth.This paper lists the current situation and limitations of the current retrieval system in the use of data structures,analyzes the problems and challenges faced by current data structures in the context of multi-core system,expatiates on the effects and significance of Lock-Free data structures on the retrieval system.Secondly,introduces the main components of the retrieval system and its main functions and data structures:RPC message queue,forward index and inverted index.Then,redesign the Lock-Free structures for RPC message queue,forward search and inverted search.Lock-free queues,restricted wait-free skip lists and restricted wait-free hash table structures are proposed by using the CAS technique,test and compare with the existing lock methods,and apply these new data structures to large-capacity data retrieval system.Finally,under the same condition of single machine,compare with current retrieval system in the same condition in the retrieval accuracy and recall rate aspects.The new retrieval system with Lock-Free structures is verified a better performance under the premise of guaranteeing the retrieval accuracy and recall of the retrieval system.