Design And Implementation Of Automatic Data Grabbing Software Based On Selenium

Automatic data grabbing refers to the automatic execution of the whole process of datacollection through the application program,so as to achieve the purpose of obtaining the target data from the webpage in batches.With the widely used of big data technology in the production and management of enterprises,data acquisition technology has been paid more attention.Currently,the most commonly used data crawling technology is web crawler.Although this kind of technology is powerful and it has high collection efficiency,it is complicated to program and all the captured web pages need to be stored.Therefore,it consumes resources and is only suitable for people with programming basis,which results in limited application scope.Therefore,we decided to design and implement a Selenium-based automatic data grabbing software.By providing data grasping techniques that are easier to grasp,we can make up for the insufficiency of crawler technology and provide more efficient and convenient data batch crawling services for more enterprises or individuals.By generating multiple types of XPath expressions for web page elements,and summarizing a set of evaluation criteria for XPath expressions,the method of locating web page elements is optimized to help the software accurately locate the target data specified by users.Then use a software-defined scripting language to save the user's data collection process as a script file.After that,we can complete batch collection of data by executing these script files.Finally,the software was tested and the test results show that the software achieves the expected goal.We hope that users can take advantage of the software developed in this paper to complete the task of tedious and monotonous bulk data collection to achieve the goal of not only improving work efficiency but also saving valuable time and effort.