Design And Implementation Of Social Network Big Data System

With the rapid development of Internet technology and the improvement of people's living standards,more and more people participate in social networks People are happy to share their information in work,study and daily life in Wechat,Twitter,Facebook and other social networks.They can not only express their views on current events and social news,but also express their feelings about trivial things in life,consumption and shopping.This results in massive social network data,which largely reflects users' interests,specialties,hobbies,views on events and emotional tendencies,etc.Efficient mining and analysis of these social network data has important applications in public opinion monitoring,event prediction,market research,product recommendation and so on.Due to the important application value of social network data,more and more people begin to design and develop systems for related research.But at present,social network data analysis system mainly focuses on a single data source,ignoring the diversity of social network data,and many systems are analyzed from a certain angle of data,which is not comprehensive enough.Therefore,it is of great significance to develop a social network big data analysis system which can analyze and mine multi-source heterogeneous dataThis thesis mainly introduces the design and implementation of social network big data analysis system.The system consists of four modules.The first module is data acquisition and fusion.The module uses the open source web crawler framework WebCollector to crawl data from Facebook and Twitter websites,then parses and preprocesses the obtained data,and represents the processed data of different social networks,and then stores them in the non-relational database Neo4j The second module is information retrieval.This module uses Lucene to realize the fast construction of full-text index and provide a variety of search interfaces,including keyword search,character search,time search,source search and their combination search.The third module is data analysis.The module uses TF-IDF weighting algorithm,K-Means clustering algorithm,CNN text classification algorithm and other algorithms to realize data mining and data analysis.The main functions are content analysis,behavior analysis,user portrait,hot topic discovery and so on.The fourth module is data visualization,which realizes the visualization display of data retrieval and analysis results through forms,histogram,line chart,map and other visualization forms to help users extract valuable data information more efficiently.The system is developed based on Java language,using B/S architecture,MVC design mode,the system adopts the development mode of front-end and back-end separation,which is convenient for later maintenance and upgrading of the system.The front-end of the system uses HTML5,Echarts,BootStrap and other technologies to visually display the results of social network data retrieval and analysis.The back-end of the system adopts the popular SpringBoot framework,the database adopts the non-relational database Neo4j,and comprehensively uses Lucene,TF-IDF,K-Means,CNN and other technologies to realize the data retrieval and analysis.