News Feed Stream System Based On Distributed Real Time Stream Computing

At the moment,Internet application development places great emphasis on traffic.The more users,the greater the value.As a typical traffic portal,the feed stream is used by more and more Internet companies and is also widely loved by Internet users.However,after the data and the number of users have risen sharply,the feed stream system will face severe challenges in many aspects such as high concurrency,throughput,and real-time performance.Therefore,it is very necessary to introduce a feed stream system that can support massive data and users while ensuring real-time performance.After researching the product architecture of many Internet head companies and current mainstream technology tools,this paper propose an end-to-end solution that uses distributed real-time stream computing to build a feed-stream system that can achieve high concurrent,high throughput and low latency.By decomposing the entire system into three major links: pre-sales,in-sales,and after-sales,news entity in the feed stream system can be treated as commodities,and the business process of this system can be better understood from the perspective of traditional commodity sales.Pre-sales includes news collection,filtering,classification,and label generation.The technologies involved include crawling,text classification,and RPC(Remote Procedure Call).In-sales means interactions between users and news.The technologies used include Spring Boot,Nginx,My SQL,and Redis.After-sales is an operating platform that includes operating data statistics,display,and negative feedback processing,using technical frameworks such as Flink and Grafana.In this paper,the control and processing of flow data in the system reflects the design idea of flow computing,so as to achieve high throughput and real-time performance of upstream and downstream data between multiple components.This article uses a distributed architecture in many places to disperse highly concurrent tasks,so that the system can make full use of cheap hardware resources to achieve efficient and highly available services.At the same time,by introducing Docker container technology and Kubernetes cluster management tools,the deployment and operation of a large number of microservices in the system have been simplified.This paper strictly follows the standardized software development process to design and implement the feed stream system,and conducts demand analysis,business splitting,feasibility analysis,that is,basic theoretical research and technology selection,specific design and implementation,and functional and performance tests,etc,in order.Use function module diagrams and architecture diagrams to explain the design ideas of the system as a whole,and use a large number of diagrams such as program flowcharts,timing diagrams,UML,and use case diagrams to explain the implementation steps of each function module in detail,and finally conducted basic functional tests and performance tests on the system,and finally delivered to the line and run normally and stably.