Design And Development Of Intelligent Indexing System For Distributed Real-time Database In Process Industries

Real-time database system is the basic data platform of production information technology in large process industries. With the rapid development of process industry, the traditional real-time database indexing techniques are unable to meet the requirements of practical application in terms of system capacity, reliability and scalability due to the relatively fixed data size, difficulties in extending the performance and lack of flexibility. Research on intelligent indexing techniques which can support mass data storage and distributed redundant backup with dynamically adjusted scale has important theoretical significance and notable practical value for the distributed technology in real-time database.In this paper, based on the characteristics and organization form of the process industry data, a kind of intelligent indexing technique for distributed real-time database was proposed. It combined the existing processes industrial monomer real-time database indexing techniques with the mass data storage indexing technology using NoSQL, realizing the self-indexing and self-adjustment function of the industrial measuring point data in a distributed environment. Users need not care about the specific storage location of the measuring point data and load for each measuring point. Problems such as the performance bottleneck of the network bandwidth and dynamic maintenance of the storage location of the measuring point are effectively solved, improving the ease of use, security, and scalability of the system.Based on the existing high-performance storage indexing techniques for real-time database, this paper creatively put forward a distributed intelligent indexing mechanism which is suitable for the measuring point data in process industries. The mechanism was constituted mainly by three techniques:index storage and query, load balancing, and multiple copies. The index storage and query technology establish a virtual token domain and map each measuring point to a token in the domain according to hash algorithm. Then it dynamically allocates jurisdiction token range for each server according to cluster load. Efficient data structures were designed to ensure that querying measuring point historical data dispersed in each server and measuring point subscription are supported after the cluster state changes. Load balancing technology uses double-loop feedback mechanism. When there is an imbalance trend in the server load, writing flow of the measuring point data is adjusted dynamically and the load of each server is adjusted in advance to avoid large amounts of data migration operation. Multiple copies technology realizes concurrent inquiry and intelligent backup of measuring point data in process industries. The service is not interrupted when the server is down and lost data can be restored. It also provides support for concurrent data query, thus ensuring the real-time and reliability. The design is the core of the distributed data reading and writing mechanism and provides a guarantee to meet the functional and non-functional indicators of the overall system.A distributed intelligent indexing prototype system was implemented based on the above mechanism. A test platform was built using4general-purpose servers to conduct a comprehensive test validation on the performance of the storage, query and load balancing under the push pressure of100,000points. The test results showed that the various functions and the overall performance of the prototype system meet the requirements of the design and application.