Research And Applications Of Shotr Text Stream Data Filtering Technology

The fast development of Internet give birth to the Big Data Era, then The rise of mobile Internet, make big data everywhere. With the development of the Big Data Era, the short text data stream becomes bigger and realtime. In the Internet age, the short text stream mainly comes from client programs and web applications, for example weibo, Twitter, tieba, instant message, query and so on. But in the era of mobile Internet, more and more mobile applications appeared, which make nearly everyone be the productor of the short text data.In the past time, the product developers mostly will be to monitor, filter and analysis the short text stream data for the purpose of protecting the user’s personal property safety, cracking down on illegal crime and public opinion analysis.At present, following the gradually in-depth development and application of data mining and machine learning techniques, more and more people are aware of the potential value of the text flow data, which has given rise to deeper and more detailed requirements and application on short text stream data filtering than the previous.In the current era of big data, text flow data filtering problem, compared with the traditional short text stream data filtering problem, significant changes have taken place in the conditions and requirements.In terms of application scenarios, now the text flow data filter problem has the character of massive and real-time. On the demand side, under the character of massive and real-time, text flow data filter requirements are becoming more and more complicated and varied. In order to satisfy and adapt to this trend, in recent years in the field of short stream data filtering, many new technologies and solutions are put forward.This paper first deep studies and introduces the popular stream processing framework, such as Storm, S4, and Puma, etc., then this article summarizes the short text data processing flow problems, at last the classified summary and in-depth analysis was carried on the key technologies involved in this paper. Secondly, based on the analysis of short stream data filtering problem, this paper ponders the key word(words)filtering technique deeply and puts forward a text flow data filtering and matching algorithm. Thirdly, based on the analysis of the specific process flow data filtering, this article discovered the text flow data filtering demand for arbitrary time granularity of data feature analysis, then design the arbitrary time granularity of data feature analysis framework. Finally, this paper applied the algorithm and framework, designed and implemented a short text stream data filtering system, and the system was fully experimented, which verified the robustness, extensibility and practicability of the system.In summary, this paper’s research work and results provide novel idea to design short text stream data filtering system and applications, and have important guidance significance and reference value to the development and innovation of short text flow data filtering technology.