Design Of Open-source Intelligence Collection System Based On Big Data

With the continuous popularization of the Internet and the arrival of the era of artificial intelligence,mobile internet technology has brought tremendous changes to human daily life.By utilizing advanced technological means,relevant staff can quickly and accurately obtain open-source intelligence information with high update speed and quantity,such as sudden public security incidents,political security incidents,and negative comments from netizens.This can effectively avoid a series of extremely expensive and time-consuming operations such as manually collecting text information and reading,analyzing,and organizing it one by one,greatly reducing time and energy,Effectively avoiding the possibility of important information omissions.To address this issue,this article investigates how to use multi-source heterogeneous data from the collected large amount of data to construct sentiment classification and thematic retrieval datasets,and then classify sentiment tendencies for news topic articles.Firstly,a themed crawler framework is built using Scrapy and Se Lenium to crawl open source intelligence information data that meets the requirements of this system,targeting stability maintenance,negative government information,and terrorism related issues.Secondly,through Jieba,Text rank and other technologies,the collected data can be processed by word segmentation,de stop words,vectorization and other data.Then,the MLP full connection neural network based on L2 regularization and Dropout optimization mechanism is constructed,and the processed data set and open source data set are trained to achieve the emotional orientation classification of news topic articles.Finally,use keyword cloud,pie chart,histogram and other visualization technologies to process and analyze the data,so as to improve the reading effect and make it easier for users to understand the central meaning of the text.Through the testing of four models: MLP,MLP+L2,MLP+Dropout,and MLP+L2+Dropout,the experimental results show that MLP+L2+Dropout is superior to the other three models,with the best model performance and the highest accuracy in sentiment analysis.It can be used for sentiment analysis of specific open source intelligence information.In addition,in order to further improve the user experience,this article also designed a simple and easy-to-use web page and mobile app,which users can easily use on both the computer and mobile devices.Through this approach,we hope to provide favorable support for open-source intelligence workers and achieve efficient processing of real-time information.In summary,this study explores how to use open-source intelligence data to construct sentiment classification and thematic search datasets for sentiment orientation classification of news topic articles.Effective data collection methods and deep learning models were adopted,and satisfactory experimental results were achieved.This study provides strong support for open-source intelligence workers to process real-time information,and is expected to further improve the accuracy and efficiency of sentiment analysis.