Research And Implementation Of Real-time Stream Data Processing System For E-community

In Web 2.0 era,Internet has become an important place for communication and interaction.As one of the communication forms,E-community has been gradually integrated into the social reality and become an inseparable part of life.E-community gives people a place,where they can freely express emotions and views,through interest of polymerization it can forms a topic or a community,and it's a key way of forming social public opinion.Monitoring and governance of the bad phenomenon in the E-community is especially important to maintain the network health.In order to achieve effective real-time monitoring and avoid the risk of negative public opinion expansion caused by the rapid spread of information,E-community's data needs real time analysis,data mining and processing,which ensures the accuracy of the analysis,what's more,ensure the timeliness of the information.The data of E-community has the characteristics of big data and real-time,which need detecting rapidly and analyzing efficiently,large-scale and real-time stream data processing technology meets the need.Based on the research and comparison of several popular stream data processing techniques,and aimed at characteristics of E-community,this paper design and implement a stream data real-time processing system,which using Storm,an open source distributed real-time computation platform.On top of distributed cluster,the system builds three layers including the data access layer,the real time processing layer,and the data storage layer to realize the real-time acquisition,analysis,processing and storage of massive data.This paper has also realized data credible validation.On one hand,message queue in the data access layer implement data retransmission and backup,which ensures the reliability of data acquisition.On the other hand,the real-time processing layer improves Storm's sliding window mechanism and ACK mechanism,provides backup and recovery measures of the untreated data,ensure security and reliable data processing.To meet the demands of E-community real-time data processing,the system achieves many functions such as top posts statistics,hot word statistics,real-time updates statistics,post content filtering,topic found,and so on.All results of data analysis and processing will be shown on front Web page.The system has been applied in university's BBS for testing.It's proved that the system can achieve real-time data acquisition,analysis and data mining,and achieve good real-time data processing results.