| With the development of the Internet,data is growing explosively.With the popularity of Social networks,Internet of things,and in-vehicle network,the unstructured and semi-structured data account for a higher and higher share of data center.Big data technology comes into being,is developing rapidly and is affecting the Internet extensively.It will cause another information revolution.Hadoop is typical of the big data technology.All kinds of companies,open source teams,research institutions are carrying out extensive research on Hadoop actively,coming up with a lot of platforms which support batch processing,stream processing and real-time processing.This paper expounds the big data real-time processing platform and describes the memory optimization scheme based on Memcached.On the issues of Memcached hotspot and single point of failure,we make a systematic analysis,and put forward the dynamic adjustment strategy based on hot spot.The research results of this paper is of important theoretical value and practical value.Theoretically,the load imbalance problem caused by hotspot is widespread in distributed systems,such as cloud computing systems which need to consider whether the utilization of CPU and memory in each node is balanced.The dynamic adjustment strategy which based on hotspot and introduced by this paper has academic value for the optimization of distributed systems.From a practical point of view,the realization of the strategy successfully accelerated our platform’s query and processing speed for large data,which proved that the strategy has practical value for building efficient big data platforms. |
