Design And Implementation Of Online Education Platform Based On Web Crawler

With the rapid development of Internet, the combination between traditional industry and Internet industry is becoming increasingly beneficial. The traditional education provides students a direct feedback while many other problems remain, for instance, the limitation of time and place, nonoptional contents of classes or a vulnerable lecturer and so on. Online education has a great advantage in terms of these cases and is possible to provide suitable contents for each student. Therefore, online education is growing rapidly. High-quality educational resources are important components of online education, especially English. However, how to acquire high-quality resources could be extremely hard, web crawler is a perfect solution.This paper derives from an actual project——Zhike English education System, which successfully gets the high-quality resources on Internet by using web crawler and the reasonable conformity of the resources makes them unique. First of all, the requirement of resources and questions were discussed as well as the prototype of web system. Secondly, Web crawler module, questions generation module and Web exhibition module were designed and developed. Functional tests on the system and each module were performed. The Grab Web crawler framework and Flask Web framework based on Python were used in this paper while different crawler algorithms and different crawler measures such as General Purpose Web Crawler, Topical Web Crawler and Deep Web Crawler were discussed. On this basis, high-quality resources were integrated and provided to users in materials and question library.At present, the project is in the experimental stage, whcih provides high-quality materials and questions. The system is stable and the expected effect is achieved.