The Design And Implement Of A Web Filtering System

Today the Internet has an increasingly important impact on various areas of human life human life, the normal life of the people has been inseparable from the Internet.However, the Internet provides people with convenient and also brings a lot of negative information. In order to provide a healthy, safe network environment, it is necessary to filter illegal web.Traditional web filtering technology includes content-based web filtering and URL-classification-based web filtering. Content-based web filtering technology can real-time filter web page based on its content and has a wide range of filtration, but it has problems of large amount of calculation and low filtration efficiency. And with the advent of the mobile Internet era, the diversity of mobile terminals makes this kind of web filtering system can not widely deployed. The URL-classification-based web filtering, which pre-analysises of the contents of web pages, reduces the user’s filter computation time. It has high filtration efficiency and high accuracy, and facilitates concentration deployment. But the timeliness and coverage of URL categorization data need to be considered, similarly the throughput of Web filtering system whether meets the specific application scenario.Based on reading a lot of references and related material, the author designed and implemented an improved web filtering system, which is able to provide efficient, versatile filtering service and gradually improve the coverage of URL classification data by automatically learning.The main works are the following:1. On the research of current web filtering technology and its application, the author analysised its advantages and disadvantages and stressed the necessity of deploying web filtering on ISP center.2. On the analysis of the demand for ISP center, the author considering the efficiency of web filtering, the availability of system, the coverage and hit rate of URL classification data, and personalization of user, proposed an improved URL-classification-based web filtering system. The system is divided into the URL classification subsystem and web filtering engine.3. On the overall design of URL categorization subsystem, the author used text intelligent classification techniques and human reviewer patern, which improving the efficiency and accuracy of URL classification.4. On the detailed design and implementation of web filtering engine, the author using MD5and red-black tree technology achieved efficient query cache, improving the efficiency of web filtering, and using dual hot standby technology achieved highly available data and system. The author also dsigned efficient query process, which supports for user-defined data and different priority; dsigned the upgrade of data, which makes the egine be able to periodically update URL classification to improve the accuracy of the filtering; dsigned feedback process of missed URL, through which to improve the coverage of URL classification.5. By the using of load balancing techniques, the author designed a distributed web filtering engine.6. On the test of key features of the web filtering engine, the author analyzed the test results and verified the feasibility of the system.