Data Stream Security Storage And Real-Time Computing Inspired By Bio-Intelligence

The rapid development of information science has significantly improved the intelligent level of production and life,in which comprehensive data collection,secure storage and efficient computing have played vital roles.As the data collection process continues,the amount of data obtained is constantly increasing and usually needs to be stored with the help of cloud computing technology.From the perspective of time series,a large number of data samples converge to form a data stream,and the information contained is time-effective,whose value will continue to decrease over time.Therefore,this dissertation studies the related theory and technology of streaming data from the aspects of security and timeliness.The ideas are inspired by biological mechanisms such as the immune system and memory system,and are to study novel solutions for different problems in streaming data.The main work consists of the following aspects.(1)Simulating the biological defense mechanism of the biological immune system and combining the distributed characteristics of cloud storage,an improved dynamic immune algorithm IDIA(Improved Dynamic Immune Algorithms)and an efficient hierarchical retrieval strategy for data samples are proposed based on immune memory mechanisms.In the process of antibody generation and dynamic update,the improved strategies such as shift mutation and random grouping are proposed to improve IDIA's antibody generation efficiency and dynamic environment adaptive ability.Aiming at the problem of cloud data storage,based on the distributed master-slave structure HDFS(Hadoop Distributed File System),this dissertation analyzes various defects in traditional cloud data security defense methods and proposes an IDIA-based cloud data security storage method,in which the computing area server efficiently extracts data samples to be calculated in the storage area.Store the high-quality data samples in the storage area temporarily to the master node of HDFS.The computing area server needs to obtain the high-quality samples in advance from the master node,and then obtain the permission to access the storage area server through the hierarchical matching process of the storage area that is based on the immune memory mechanism.This process can quickly match "self" data,and at the same time can effectively identify "non-self" data samples to achieve security defense.(2)Inspired by the ability of the immune memory mechanism to efficiently process dynamic data streams,the brain-like memory systems including hierarchical memory mechanism and temporal memory mechanism are further explored.The hierarchical memory mechanism divides the information in the brain into different layers by the amount of information traces.This dissertation divides it into three layers: short-term memory layer,long-term memory layer,and permanent memory layer.Then by modeling the memory,recall and forgetting mechanism,the importance of data samples is quantitatively described,and the dynamic migration of data samples between different layers are realized.In temporal memory mechanism,the historical experience knowledge can be used as references for predictions of the following moments by continuously memorizing,strengthening and reproducing.(3)This dissertation divides the abnormal problems into explicit exceptions and implicit exceptions according to how it is formed.Explicit anomalies are usually caused by external physical reasons such as noise interference during transmission,network instability,and equipment aging,while implicit anomalies are represented by changes in the internal data distribution.Combined with the practical application background of the process industry,the explicit and implicit exceptions in the data stream are studied through batch processing and stream processing respectively.Batch processing mainly addresses explicit anomalies such as missing values,outliers,noise,and redundancy in real-time data streams.The significant difference between streaming data and traditional data processing is that the former pursues approximate solutions and takes timeliness as the main indicator,while the latter uses precise solutions as the main indicator and lacks consideration of the timeliness of the processing results.The hierarchical memory network designed in this work is an approximate solution to the abnormal problem based on sample replacement,which is suitable for online systems with strict timeliness requirements.(4)Data stream is a real-time perception of the dynamic environment,which also includes streaming processing way apart from batch processing.Stream processing uses a single data sample as the processing unit.Among them,there are widespread implicit exceptions problems represented by concept drift.Therefore,this work is inspired by the temporal memory mechanism,designing a new type of abnormal numerical detector based on sparse discrete representation,and constructing a temporal memory and learning network.The network uses historical empirical knowledge to predict the sparse discrete representation of unknown data at the next moment and compares it with the actual representation at the next moment to complete the anomaly detection.The decision matrix of the dynamic data stream is defined,which helps to identify the concept drift phenomenon from the dynamic data stream with noise,and prompts the detector to update in time to adapt to the new environment.The construction of the decision matrix is based on the trend of concept drift and the disorder of random noise.The change in real-time production requirements is reflected in the data stream as a concept drift.The timely monitoring and updating of the encoder is to meet the needs of customization.Finally,this dissertation summarizes all the work,analyzes the shortcomings in the current research,and looks forward to the problems that need to be further solved.