Design And Implementation Of Crawler Subsystem Of Alipay Merchant Risk Monitoring System

With the rapid development of electronic commerce, online trading volume isincreasing, the online banking system can not support the requirements of customersto pay alone, so a series of third-party payment companies are springing up. Theynot only contain major banks’ payment interfaces, but also rich in many applicationsto meet the needs of customers’ online payment. The third-party payment platformallow e-commerce websites to provide customers with the online payment serviceby payment interface.This paper analyzes the specific circumstances of Alipay online paymentplatform, we can see that the platform just offers online funds services, but thespecific content of the transaction is not known, so a lot of websites do illegaltransactions online and seriously undermine online payment environment.We analyse the scale of the current access to e-commerce websites to find areasonable set of merchant risk control program and establish a merchant riskmonitoring system. The system also provides the administrator back-end userinterface, and allows manual intervention to validate suspect merchants which thesystem finds, to verify and ensure that final rectification is indeed done on illegalmerchants.Merchant risk monitoring and control system is divided into crawler subsystemand audit task subsystem. This article shows a detailed needs analysis and design ofsome functions in crawler subsystem, including basic data maintained, basic dataloading, load balancing of distributed environment, web address distribution andde-emphasis, and text analysis results processing. According to the current businessvolume, we design a reasonable physical deployment architecture. Basic datacontains keywords, merchant sift strategies and scenarios. The basic data is used forselection of high-risk merchant sites and analysis website content. We use theHash-based task scheduling mechanism to achieve distributed crawler loadbalancing, and we use BloomFilter data structure to achieve an efficient web addressde-emphasis mechanism. After keywords analysis, suspect text will be packaged asverification task and distributed to audit task subsystem.The system has been running on line from December20,2011. The actualoperation verifys the the real value of theoretical study and design and we can seethat the monitoring of the merchants played a relatively good results.